I must admit, it's been a while since I've been interested in learning a new programming language. 10 years ago, I would have jumped into a new language in a heartbeat. Now, I tend to ask the question, why?
Let's be honest with ourselves as we get older, we have less time to learn new things. We have more responsibilities, and we have to be more selective with our time. So, when I heard about Rust, I was skeptical. I thought, "Why should I learn this language? What does it offer that I don't already have?".
But then I started to hear more and more about Rust. Strangely enough, from watching videos on Youtube, although the first time I heard about Rust, was at work, one of my colleagues mentioned it as something we need to consider for a few of our services. I heard about its performance, its safety, and its concurrency. I heard about how it was being used in production systems at companies like Mozilla, Dropbox and soon the Linux kernel. I heard about how it was being used to build the next generation of web applications.
Of course my first thought was "So it sounds like the do everything language, bigger, stronger, faster, the future of programming languages".
I started at university using Java, that was "the" language to learn. All I ever heard from my professors was "All the jobs are in Java", "Java runs on everything". One lecturer was a Python guy and he really didn't like Java, he actually put me onto Python. Strangely enough I only really understood programming once I started using Python. I decided to have a play around with Python on a summer project after my first year at university. I'll be frank, I have no idea how I passed my my first few programming papers at university, I was lost in Java. I mean I kind of new what I was doing but it didn't quite click, something was missing.
I have no idea why but Python just made sense, maybe it's because I was using it in the context of a project that interested me. It was a simple parser to interact with the "LastFM" API, a tiny project. I was able to get it working so fast, and I could see the results, the web made sense, API's everything just made sense. Best of all? I didn't need to compile, I just wrote the code and ran it. I could see the results instantly. I was hooked.
I ended up loving Java, and once I graduated, I actually tried to get a Job at a Company that developed web applications in Java. I didn't do well in the interview, I was too nervous. I ended up getting a job at a company that developed a fintech application in dotnet (C#). Arrrgh windows! I really didn't want to work in windows. But I really needed the experience.
I ended up enjoying it, some of my best working memories are from that job. The senior developers were great, actually everyone was amazing, they taught me a lot. I learned a lot about software engineering, architecture, working in a team and most importantly, that I didn't know anything! I am a very shy person and so just being able to work with people I don't know in itself was an achievement.
Most of my time in the dotnet world was actually spend learning about the dotnet world, and it's a big world. I walked and talked dotnet for 7 years. Then I discovered the world of Javascript.
An important thing to note is that if there are a group of people that hate Java, it's developers that work with dotnet. I don't know why, but they do. I think it's because dotnet and Java are so damn similar. Neither will admit it, but really you are using the same damn language get over it. In those years at the company one language that was usually never mentioned in a positive light was Javascript. To this day, I'm not sure why, I mean the web now is just javascript. To be fair the web application that existed when I was employed was still webforms which is really old school, so Javascript didn't really fit in...
Towards the end of my time at the fintec company, I was interested in something new, I wanted to see what this Javascript was all about. After all, how can people hate something so much? I'm not a frontend developer, I can develop across the stack, but I prefer service side, I just enjoy solving complex problems, distributed systems, scale, networking, databases, queues, streams, you name it. Don't get me wrong I love seeing things happen in the UI and I can definitely develop a frontend and enjoy it, but my happy place is the server...
Funnily enough, the principal architect at the company, mentioned NodeJS to me when he was talking about a new project he was looking into for his startup. He mentioned the "MEAN" stack and microservices, it sounded so cool, one Language across the stack? What sort of black magic is this? I had to see. I really respected this guy, and to this day do, he is an unbelievable developer and architect. I was sold, I had to see what this was all about.
Apart from a cool acronym, the MEAN stack is essentially a full stack development framework that uses MongoDB, Express.js, Angular and Node.js. Javascript all the way through the stack. I tend to learn by doing, I try, fail, try, fail and try again, I basically learn by doing. I needed a project though, I have to build something that interests me. My Python journey came to mind, and so "what would I do?". It turns out, Learning management system, I still don't know why, but I was so interested in learning management systems, but there it is, the project...
I started building a learning management system, I was so excited. I had to learn Angular, MongoDB and NodeJS. I was in heaven, I was learning so much. I was able to build a full stack application in a matter of weeks. I was hooked. I ended up building a "full" learning management system, it was amazing. I learned so much about the web, about how things work, about how to build applications. It was amazing. Alright, not a full learning management system, but it was a start. I even managed to build a simple video editor into the application. I could prototype and develop so fast, no building of code each time and using nodemon, development was a breeze. I was in heaven.
I never really noticed the scale issues that NodeJS can suffer from, to be fair, I think most people that like to point out "speed" as an issue with NodeJS, aren't building applications that need the "speed" of dotnet, Java, C++, C or any other language. This is of course from someone that hasn't worked at Netflix/Dropbox/Facebook, or a like with millions of users so take that with a grain of salt. I can see the advantages of any language really, and the right tool for the job is my moto... I find a lot can be said about the downsides of NodeJS but I find it great to develop applications. In fact the fintec company I worked for would have benefited a lot from using NodeJS for some of their web services, it would have forced them to think differently about their architecture and developer productivity. Of course Typescript is a nicer addition to the language which definitely helps those who don't dynamic languages.
I ended up leaving the company and trying my own "start up", if you can call what I did trying to start a company. I really just learnt more about NodeJS, software development, tooling, CI/CD, AWS and Docker. I was then contacted by a recruiter about some Jobs, a software architect Job. Well, that is interesting, I thought. Towards the end of my time at the fintec company, I was development manager, I actually really enjoyed it, but they were really slow to adopt anything really and I didn't have enough "pull" to really make the changes I wanted. But as software architect, I had the power to drive change! Well I thought so anyway...
I ended up getting the job, I interviewed well, the company is great (I am still there) and the people are amazing. I was able to drive change, I was able to build a team, my manager is awesome and while the company isn't a "software company", I love it. We try to step out of our comfort zone and we are slowly becoming more software focused.
Up to this point, I wasn't too worried about speed of a single instance of a service. I mean, just add some more instances and scale out. We used docker containers so just keep scaling, this works well for our use cases. Fast prototyping, development, deployment and it's simple, anyone can build a new service and deploy it. Then we found that as we grew and branched out into new areas, well, resources use, speed, memory management are more important. We don't have millions, or thousands of users, we have a few hundred active users if that using our web applications, but we process a lot of data, at least for a company in New Zealand, and there are a good amount of transactions through the various services. We probably have too many services at this point but we are working on that. One service in particular was starting to show signs of stress, it was a service that processed a lot of data and had a lot of transactions. It was a service that we had built in NodeJS, and it was starting to show its age. It basically loaded billions of events into memory and processed them. We could scale it out, but each instance really needed to be fast, really fast.
We have a finite window to process the data and so we need to be quick and doing this out of memory makes sense, we have memory, so let's use it, but we also need to think more about processing the data in a more efficient way.
Step in Rust...
The journey is coming soon!