Research projects in a small company

How do you take time to do pure research on new technologies in a small company?  This is not a rhetorical question.  Small business teams rarely have the money to justify spending months on research.

It is usually a case of build what works and run with it, so when do you get the time to do that research?  Without ongoing research into new technologies and designs you have to find a way to fund those activities. Building a product by growing things in place is a good way to accrue a lot of technical debt that eventually leads to a fragile design or a complete rewrite.

How do other companies handle this?

Ok, this is my personal answer to the question; Your Mileage May Vary (YMMV). It may or may not be correct for everyone, or every situation.  I want to share parts of how I came to this conclusion in the hopes that it may help other small companies.

I have always believed that any product without continuous development is dead.  How can a product innovate without new work being done on it?  Only through investigation and application of technology can a company learn to innovate.  Otherwise you are just playing catch up with others in your market and telling the developers to do what you competitor just did.  Sure, maybe you improve it a bit, add some polish.  But without looking way ahead of the pack you will never leap frog anyone.

In large companies they usually handle this by having multiple teams.  Team A is on the current release, and Team B is looking at the next major.  Then they flip flop when Team B is ready for production Team A then goes into looking at the next major release.  Small businesses can’t afford to employ this strategy.

Some companies do this by having a long release cycle.  They release 1.0, then spend 2 years getting to 2.0, etc.  This leads to long cycles of no improvement in the product, but for some industries this is acceptable.

Most small companies iterate over their design making small improvements and adding customer features.  At some point it seems like “enough” changes and they rev the major.  Some small companies ignore the problem entirely and call everything Product 2009, Product 2010.  Basically telling you that they are all the same, but with some form of improvement over time.

There is nothing wrong with any of these strategies, but they don’t work in our scenario.  We have large research work to do in some fields that just can’t be iterated into the core product at all.  They are literally all or nothing scenarios. You could spend a year of effort to build it into the core engine only to discover it doesn’t work well.

New ideas are sometimes too big

In our case we build a great dot net database engine.  The architecture is quite different from SQL Server, in that we are 100% managed code.  But do we really take advantage of everything that managed code has to offer?

There are newer data structures and practices in Computer Science for dealing with very large data structures.  Can we employ some of them for our indexes?  Dot Net 4 now offers memory mapped files for the first time, could that help our sort and indexing?

What about some of the new parallel abilities in Dot Net 4?  How can we leverage them without spending a huge amount of time integrating them in the main database?

Our SQL parser does not implement a grammar parser like BNF.  The entire parser is built around a pcode system that probably worked great in VistaDB 2 under Pascal, but is problematic under .Net to expand easily.  Would implementing Oslo or one of the other domain specific language (DSL) parsers allow us to be faster and more flexible with SQL dialects?  Could we, for example, parse MySql syntax as well?  Under the hood we are not SQL based at all, what if we just exposed a pure LINQ syntax and dropped the SQL parser entirely?

Trying to build and implement any one of these ideas into the main database engine could take a very long time with no yield at the end.  How can we work on these theories and mitigate our risk?

Design a new product around a single idea or technology

What if you took and built a new product around specific ideas or new technologies?  Nothing real fancy, not a general purpose answer for every scenario, just a very specific niche.  Don’t try to implement it in your main product.  Just look at the technology and where could it go, how would you build it, apply it, sell it?  Do you think you could learn lessons from that specific application that would help you better serve your larger customer base later?

Could we use a domain specific language (DSL) to implement SQL dialects for a Data Migration API first, before going to the main database engine?  Could we build a way to generate large quantities of test data through an API as a way of testing load and index systems?

Could we use the Dot Net 4 parallel extensions to perform faster indexes for a specific type of index first, like full text search?  Could using other indexing strategies lead to better search performance, indexing speed, or both?

Could we use different technologies for ETL (Extend Transform Load) of database data into VistaDB?  Could that provide a way of performing bulk load, or transformation services?

These ideas would allow us to work in isolation without fear of breaking the main engine, but still move forward on research.  And many of these small niches fill product needs for people today, why not offer it to them?

But it’s not VistaDB…

Eventually much of the technology may find its way back into VistaDB, but not directly.  Each of these technologies will be stand alone tools or products.  Each needs to be applied against specific use scenarios to prove or disprove ideas. In some cases they may work fantastic for their applied niche, and never make it back into the core product because they don’t work well in the general purpose case.  But by doing the research we will know the answers, without wasting a lot of time ripping apart our core product to test theories.

These new technologies have different potential customers, and needed their own name.  As a result, we are building new websites specific to some of these ideas.  Some of them will be released as stand alone products or APIs for specialty use scenarios.

So how do you brand that other “stuff”.  It isn’t VistaDB, it is something different.  I didn’t like the idea of publishing it under the VistaDB brand name as I think it would confuse users about what VistaDB is as a product (100% managed SQL database engine, just to be clear).  All this other stuff is new technology, not constrained around our core product.

Would it all be “Brought to you by VistaDB”?  That would seem sort of misleading because then people might think it was VistaDB under the hood of the tool, in most of these cases it will not be anything like a full database engine.  In fact most of these niche users probably won’t need a database at all, maybe they only need this niche product.  Giving it the full VistaDB name might scare these users off because they don’t want a full database.

We eventually decided to change the company name to reflect the fact that we will have more than one brand of product.  A new site is being built to reflect the idea that we are a technology entity producing various products.

Infinite Codex, Inc is born

We are going to change the company name to Infinite Codex, Inc. signifying that we can write an unlimited number of programs that are all related by application of technology.

I think naming the company and the main product the same thing was a mistake (several people advised me against it at the time. Sorry, will listen to you more in the future!).  The company name needs to be independent of any single product brand it produces in order to be able to grow and expand into other markets.

VistaDB will continue to be the core of the products we produce, but we are going to start producing other smaller systems as well.

Research, but with specific niche products

Either it will be brilliant, or totally insane; only time will tell.  This is how we, as a small company, are attempting to pay for the various research we need to do in order to expand.  Most of them are not directly related to VistaDB as a product, they are either internal technologies, or external tools that could apply to almost any database.

You can’t sell pure research without it having an application.  That application will drive your research much further than pure abstract ideas ever can on their own, and as long as you are filling a niche you can derive income from that niche at the same time.

All of the new technologies are being built with an eye towards making us a stronger company within the database market  in general, but at the same time expanding our market into other niches.  If we can better understand single technologies, then we can better incorporate them into our main database product over time.  But if along the way we become more widely known for implementing really good developer tools and APIs, we will have a larger market in total.

This all seems familiar to me

At my last company it was this ability to take our core product and apply it to so many different niche applications that eventually led to outside interest in purchasing the company.  Having that core technology demonstrated across several different niche applications made us more attractive, and proved just how flexible the technology was for almost any scenario.

The end goal of all of this research is to lead us to a stronger next major version of VistaDB, and to look towards the future.  I am sure many of these technologies will find their way into incremental updates as well, but I am already thinking of the next major engine structure.

In order for us to go where I want with the database engine we have probably two years of research ahead of us, a little longer if we productize these ideas along the way.  But, I think in the end it will help the company into new markets, new partners, and fund our growth over the long term.

I am not just thinking of any one database vendor, or application.  I am thinking more in terms of where is the Dot Net platform going, the growing NOSQL movement, and what will that mean for database apps 7-10 years from now.

Expect new announcements of technologies and components in the next few weeks.  We are all pretty excited to have so much new stuff prepping for launch, I hope you are too.