The Unusual Project Management System Notion Uses To Build AI
17:52 Share

Notion's AI team is pretty crazy. The first version of their AI dropped just last year and now they're on track to basically build the AI everything tool. But how do you actually build software in this new space if no one quite knows what it actually can do?

A few weeks ago, I had a chance to sit down with Shir, the AI engineering lead for Notion. If you're curious how an expert writes prompts and how project management has changed for this new kind of technology, then this one is for you. Shir, can you maybe tell us what does a typical day in the life of an AI engineer at Notion look like? Yeah, um...

So building AI products is very similar to building other products and also very different. The lifecycle of an AI feature, given that the technology is so new and we don't know what it can and cannot do, looks a little bit different where we prototype a bunch of stuff first just to figure out, can AI even do this? And then once we get through that phase, then we'll go back, design, think about

think about the product, think about the user experience, which is maybe the opposite of a lot of other projects where you first...

design it, think through what the user experience is going to be and build it. We like build it first, figure out what it can do and then design it afterwards. Which is kind of fun. Yeah, that sounds really cool to like take another whole development process and put it on its head. Yeah, it's a little more fun too. Do you have any, you know, like ever any like of these weird moments where it's like, oh, well, we didn't expect

expect this at all to work like this or any big frustrations? So many. Well, I mean, we've learned the only thing you guys have seen is, well, we have shipped. We actually built probably dozens of different features that just did not work, that we never shipped.

But one, actually the origins of Q&A, we started by integrating it into the regular search experience. But then we discovered that the new technology is actually better at answering questions than it is at like doing a keyword search. And so when we integrated it into keyword search,

It was like so-so, but it wasn't, it wasn't life changing. Yeah. But then when you started asking questions, all of a sudden, boom, it worked. And so it was like a, um, and an example of something a little bit surprising. Yeah. I,

I fully agree, like this question-answer moment, because it solves one of the biggest problems of knowledge management, where people create all that cool documentation, and then it's saved in some cloud storage, and SharePoint is the good documentation goes to die, because no one knows how to find it. And now this is, I think, really like a light bulb moment, when for the first time you ask Ocean AI, can you... And the technology for it is pretty cool, too. You can take...

Basically under the hood, what we're doing is taking a piece of text, translating it into a number, and then taking your question and translating that into a number. And then if those numbers are close together, it means that they're like similar content. And that's just like a very, very powerful, powerful feature for like question answering, but also for other content.

Oh, yes. I still remember like in the early days of AI when, you know, everything was completely new and you tried to figure it out for the first time. I mean, I'm not an engineer, so my knowledge is very limited. But like the little things that I don't know is like, OK, you know, string comparisons. The word needs to match the word. So like the computer doesn't understand that hi is the same as hello. And just to imagine how much, you know, like engineering goes into teaching code.

that hi is hello, that you now can do that. I think that's- - Yeah, now you can do it with like a single API call. But if you think about like old school search system, like I worked at Google on Google search before and you had these like massive systems to figure out if two words are synonym.

And you don't need that anymore with embeddings and AI. Yeah, that's really cool. And it's maybe a good transition because you just mentioned you worked before at Google, right? So maybe tell us a little bit more. What made you join Notion in the first place? Yeah. So it's actually kind of a funny story. It was at Google and then Waymo and I was trying to figure out what I wanted to do next. And the only thing I knew was that I wanted to go to a smaller company.

And so Notion reached out because I was looking and I was like, ah, you're too big. I want to go to like a 10 person startup. Notion was like 200 people at the time. But at the time I hadn't heard of Notion. So I was like, well, let me just like play with this tool and see what I think. But I'm not going to interview. Long story short, I created a job tracker inside of Notion. And I was like,

This is amazing. And loved the product so much that I reached back out and I was like, well, okay, maybe I'll interview now. And then I joined. That's really cool. And maybe can you share a little bit with us, like, you know, how were these like first days at Notion? What was the experience of becoming a Notino for the first time? Yeah. I think the thing that was most...

about joining Notion was just how well documented everything was internally. So where I came from before, things were documented, but it was mostly like Google Docs, a little bit spread out, and there wasn't a culture around keeping things productive, keeping things organized. And what was really fun to see at Notion is that you are joining a company full of Notion productivity enthusiasts. Yes. And so...

When I first joined, I joined as a manager and the people that were reporting to me created one-on-one ducks for us to keep up. And I just, I thought it was so cool because it was just not something that engineers at other places do. Yeah.

And so that was really nice. Yeah, nice. I mean, it's also very meta, right? The cool thing about Notion being like a product that helps facilitate collaboration and then building that tool which you can use it constantly. So yeah, just have like a different pulse on it compared to something where you're building something but the users are different people all the time. Yeah, we dog food our own product, F&A. Which means sometimes it's a little bit more broken than everyone else. But I think that's a huge...

like flywheel for us. So we, we build the features that we want. Yeah. Oh yeah. That's really cool. Share the notion of AI team, right? You ship improvements incredibly fast. I can't believe it. It's just like a little bit more than a year that we first had the first integrations. Maybe, can you tell us a little bit about how you're able to, you know, develop the product and deliver such meaningful improvements so quickly? Yeah. Um, so I obviously have to say the notion is a tool. How does this move pretty quickly? Um,

If you look at kind of our internal notion set up and look at just our process and meeting structure, it looks,

very different today than it did even just a month ago or four months ago, a year ago. And that's kind of using the flexibility of Notion to change the process so that it never gets in our way. Yeah. Um, so that's one piece of it. Um, another piece of it is, uh, fail as fast as we possibly can. So we, we build and prototype first. Um, we try and de-risk, um, that whatever we're trying to build works as soon as possible. Yeah.

Um, and then that actually saves us a lot of time because instead of investing funds in a project, we invest like a couple of days to see if it's viable. Yeah. Um, and then we cancel a lot of projects that we like don't think are going to work. Um, and then we also try not to fall into the sunk cost fallacy. So kind of counterintuitively, um,

because we just because we've invested in something doesn't mean we have to see it follow through and so it might feel like you're wasting a lot of time building things that you're not shipping but actually you waste more time building something going all the way to the end shipping it and then saying that it didn't work and so yeah I think Phil

fail fast yeah be really flexible yeah um try lots of different things yeah that's cool and so i assume you do project management in notion probably uh is there like you know a specific philosophy that you follow there right do you do agile like some sort of uh you know like

common methodology or like, how can I imagine, right? If I'm an engineer that joins your team tomorrow, how would you breathe me on the way you do projects? Um, so my personal opinion here is that your project management has to match the phase of the project that you're in. Um, there isn't, there isn't always a one size fits all. Like sometimes when you're maybe at the end of a project cycle, you're about to ship, you need a very structured task management system, um,

But if you're in a really early exploratory phase, creating concrete tasks to track everything and like building like very specific sprints actually can get in the way. And so at the very beginning of the project,

There's like a single doc, some bullet points. And maybe some checklist items. But it's very, very open because you need to be that open at the beginning. And then towards the end, we have a task database. We've got milestones in the database. We're like checking things off of a list. We have regular stand-ups and bug bashes. But I think it's critical to match the process to the lifecycle of the project. Yeah.

That's like fascinating. I fully agree about like having, making sure that you don't add too much structure too quickly. It's a problem I also see like a lot of times with process or something like that, that you, you know, you sit down and you map it all out in a vacuum and you very rarely, in particular with like, you know, these new explorative things where you don't know what the end result looks like, like,

how big are the chances that you're going to map out the project in the beginning. And I think especially in AI, like our first versions of Q&A, like I said, were search integrations. And we didn't,

conceive of building a chat interface at that time. Right. And so if we had tried to roadmap it all out, we would have spent probably a couple of days or maybe even weeks. Yeah. Roadmapping out the wrong thing. Um, and so, uh, it's a lot, I like the iterative style of you kind of only build out the train tracks that you need. Yeah. Yeah. As far as I can see at the moment. Right. Exactly. Exactly. That's really cool. Uh,

Maybe zooming out from like the, maybe it is also not an AI answer, but like zooming out right in the whole things that you use AI for in your day-to-day. Do you have any, you know, favorite use cases that stand out where you're just like, you know, this is so fun and playful? Yeah. So one of the things, I'm in the middle of our performance reviews right now. We're giving everybody on my team feedback. And so I was able to do this last cycle. We do it every six months at the moment.

What I do is I bring in all of the feedback that I get for each person, the self-assessment that they write, and then I have Notion AI generate the first version for me of their strengths and weaknesses based on all the feedback. That takes a minute, right? Before AI, it used to take me like

30 minutes or even hours if the stuff was all over the place. And then once I have that, I can actually give...

use my brain to give much better feedback because I'm not so tired from like aggregating everything. And then last cycle, um, my direct reports, they, they told me that it was like the best performance reviews they've ever gotten. And I was like, well, this took me the least amount of time that I've ever, uh, uh, and so I think that kind of when AI can do all the manual work for you, you can use your brain to do the creative work. And that's where there's a lot of value. Yeah. I like that. Yeah.

And I think this is like also one of the huge unlocks that you only start to understand once you work with AI. Because previously, right, like unstructured data, it was just not something that you couldn't work with technology. There's just no script that you can write. Nothing will help you digest this like freeform essay. But now with large language models, you can, you know, just a few clicks, turn it into the structure that you need. One of my favorite things to do with AI as well is I take like,

a meeting notes doc or even my raw thoughts yeah and I ask AI to turn it into a table yeah um and you'd be surprised at how good it is at figuring out what structure yes it should be yes um or give me a set of options yeah it's like great yeah and so formatting restructuring is also great got

Yeah, I think that's the secret, right? Like thinking less about like generative AI as, you know, like create something for me because like by necessity, by the way the models work, it has to be average, right? Because it predicts based on that. So like it won't come up with something novel, but then using it instead of as an analytical set, right? And like, you know, doing that part of the work, that's such a huge lift.

Yeah. And like taking information and making it bigger or smaller. Like if I have a big, a massive document with lots of different things in it and I want to generate an announcement to send out about this. Yeah. That like quick summary that I can send out. Yeah. Yeah.

is great yeah so like basically the it's like a resizer yeah for text yeah i love that uh so maybe as someone right who has this like front row seat to to ai and the black box that can often be do you have any tips you know for for someone who's you know trying to get ai to do what they want but they're sort of frustrated because the outputs don't really match it and again right it is a black box so it's hard to know you know with your experience do you have any any tips for that

So the things AI is particularly good at synthesizing, so like summarizing, bringing things together

kind of restructuring. So one format into another search. So like finding, you know, these two pieces of content are good together. And those are kind of a notion about good Lego blocks. Yeah. And so as you're thinking about what's a really good use case for AI, anything that involves revolves around those will probably work quite well. Yeah.

Anything that you're trying to do that's much more complicated. It's still possible. Yeah But takes a little bit more work and so I think first putting yourself in a mental model of okay Is this a thing that AI is naturally gonna be really fast at or is this a thing that I need to like? Work with it massage it. Yeah And could I think a lot of people get frustrated when they expect it to work, but actually actually it's in that second Yeah, category. Yeah

And so sending your expectations well. And then breaking the problem down. So if you're trying to do something really, really complicated, like, for example, I need to generate a report the other day, synthesizing information from lots of different sources. I didn't ask AI to do it

in one go because it was not going to succeed. But I asked it to like, okay, first generate an outline of all the different things. And then for each section of the outline, I'll ask a specific prompt to

to find the information there. And then once I get the information there, it'll ask another prompt to structure it in the way that I want. And so if you can break the problem down into those components of synthesize, search, restructure, and then just kind of go iteratively, I think that's the best. At least for now, until the models get better.

Yeah, that's true. But I see that as well. Like, for example, just for like a client where we build something to help them repurpose long form content. And in order to do that, right, like there are 16 prompts that are trained and used and that are output from the previous model because like it's similar like if you manage person, right? Like if you tell them, well, like do this big job and they've never done it before, like they will probably fail at it. But if you think like a manager, think like, you know, this was like the smartest intern that I ever hired, but they have no idea what they're supposed to do. Right, right.

how do I walk them through that? Yeah, totally. Same as humans. You break the problem down into pieces. It's much easier to do. Who knew? Who knew? Best practices of work also apply to AI. Yeah. That's really cool. Maybe one last question before we wrap it up. You know,

I'm not sure, but maybe friends or relatives often ask, like, oh, I am supposed to write prompts with AI. I'm not really used to that. Do you have any tips for them? What would you say to them if you want to get better at writing prompts that get you the precise output that you're looking for? Yeah, I'd say if you want precision, be precise. So if you ask something really vague, actually, your intern example, if you ask something really vague...

AI can go many different directions, so be much more specific.

And then break it down. I think if you ask for too much all at once, sometimes the model is going to handle it, sometimes they can't. And so the more precise, the more specific you are, the better it'll be. And then also know what AI is not going to do well. And don't try and fit a square peg into a rat hole. And stick to, at least for now, the synthesis, the restructuring and searching. Hopefully more. Yeah.

Amazing. Well, thank you so much for taking the time. This was super interesting. I can't wait to see what Notion and Notion AI is building next. And yeah, hope we have another conversation soon. Sounds great. So much for my interview with Cher. But Notion AI isn't the only area with big improvements this year. Notion shipped more than 60 individual features and updates. And I created a video that goes over every single one of them. So to make sure you don't miss anything important, check this one out next. I'll see you in a few seconds.