Our Growing Pains Building Enterprise Software for Video Game Arenas, LAN Centers and Virtual Reality (VR) or eSports Venues.

Interestingly enough, after being a terrible blogger over the years, somehow my blog here still sees nearly 1000 unique visits each month.  I am sure there are not many other resources for researching starting a gaming center and good old Google probably pushes a lot of keyword seekers my way.

For those that follow regularly you know that I started my first store 12+ years ago in 2004 which has always concentrated on events and eSports.  Before eSports was cool, we held tournaments monthly and in some years even weekly or more.

In late 2013 we released the first version of our software for tracking in-game stats and rewarding and ranking players in centers called ggCircuit.  We quickly grew to support hundreds of centers around the world but the software we all used to run our stores was archaic and no longer served our needs.

Here is an example of how we built our software originally to work with third party management software running in the local centers:

how-ggcircuit-was-built

So in late 2015 and early 2016 we began building our own solution from the ground up which we call ggLeap.  Instead of following the traditional model of local servers doing all of the heavy lifting, we wanted a cloud-based solution to remove the need of highly trained technicians setting up stores.  Not to mention the hours of work it takes to make everything work in a center environment.

By summer of 2016 we had a pretty good working prototype and I sacrificed my two eBash centers in July to move from our old software to the new ggLeap solution.  By sacrificing I can show you the slump in sales year-over-year that we have experience both because my focus is on the software primarily but also because our customers have suffered through all of the errors running beta software.

We also installed it and ran our set-up of 200+ stations for Gen Con in August.  By using our live installations as the sacrificial lambs we gained months of testing in weeks and sometimes days and turned our development team up to turbo speeds fixing issues for our real world environment.

Here is how ggLeap now works:
how-ggleap-changes-things

The great thing about accelerating our development cycle for eBash was that we were ready when we received a phone call from UCI eSports who had heard about the development of our solution and was 3 weeks from launching their arena.  They had tried all other available solutions and they had all failed network security requirements on campus.  Again, at the expense of our dev team’s time and our increased cost we pushed hard, flew out to their grand opening week, helped set-up our software as well as just helped with general set-up and they launched right on time.

At this point we were really drained financially, mentally and physically.  So we put out the call to our network of centers to join us on the trip and become pioneers in the process.  By contributing funds that would be stored as future credit on license fees they could start using the software early before our full release.  Throughout October we brought on just under 100 new centers.

Since we now had a great test bed, we ended the pioneer program and just allowed centers to sign up for early access if they wanted to run the software before release.  Since November we have brought on another nice size group of centers.

Then in December we were approached by Alienware to bid on a contract to run our software on their new showcases being built at 50 Best Buy locations in the US.  This was exactly what we needed.  The chance to ramp up our development cycle building features that were both applicable to their project but also important to get us closer to full release for game centers.

Over the holidays our team work furiously, we were awarded the contract with only 3 weeks before the first test launch.  In the middle of this we also had a few of us traveling to CES 2017 in a partnership with EBlue to show off our software alongside their awesome eSports hardware solutions.

ggcircuit-and-eblue-at-ces-2017

ggCircuit and Eblue team showing off our eSports arena set-up at CES 2017

January was a blur, I was flying everywhere and so were many members of our team.  Not only at CES for a week, but check this travel schedule out:

1/2 – Jmac flies to MN for BB
1/2 – Zack/Dustin fly to LV for CES
1/3 – Zack/Dustin set-up CES, Jmac testing at BB
1/4 – Jmac and Mark fly to LV to join us at CES
1/5 through 1/9 – CES and travel back home
1/11 – Zack and Dustin fly to MN for BB
1/12 – 1/14 – Zack and Dustin set-up first BB install in Mall of America
1/17 – 1/19 – Dustin back in MN for work on installer
1/22 – 1/25 – Dustin/Zack and Mark in TX for second install at BB

Keep in mind that this entire time the development team is working 12-16 hour days and most weeks 7 days per week.

ggcircuit-and-ggleap-with-alienware-at-best-buy

ggLeap and ggCircuit at the Alienware installation for Best Buy Mall of America

Our very first hiccup came the week of 1/22, where we had issues with our messaging service.  This is not messaging like chat, this is a service that takes all of our data between clients and servers in the massive network.  The third party we were using was starting to falter and we had just used them to save time thinking we would go back and setup our own later.  Well, later became that week as we began our transition to solve the 4 hour outage we had with that company’s services.

This is such a crucial part of our system because this messaging service provides about 20 links between processes that run on our servers with EVERY single location we run.  So…. imagine for easy math we have 100 locations we support, that is 20 x 100 = 2000 simultaneous links with processors.

Which brings us to last night, February 3rd.  This week we have added more centers as well as added the remaining BB installation locations and it all came to a stop as the system was overwhelmed with connections (still running on the third party messaging platform, as our devs were planning the move to our solution this coming Tuesday and the last two weeks of updates were laying the foundation to make the move next week).

So for about 3 hours last night again the servers were extremely spotty and down for long periods of time again.  I am not sure that it is clear, but trust me that the pressure on us for supporting a network of centers is like a reverse pyramid marketing program.  Think about it this way:

– Every center running our software has tons of customers that are upset when things don’t work and they take it out on employees, lets say 100 points of pressure for easy math or “100 PP”
– Employees don’t like it when customers are mad, so they then put the pressure on managers/owners, let’s say 3 employees per store x 100 PP = 300 PP
– Managers and owners then put that pressure on the ggLeap support staff, here is where it gets scary, 200 owners x 300 PP = 60,000 PP
– ggLeap support staff are then passing that along to the dev team, 4 support staff x 60,000 PP = 240,000 PP
– ggLeap devs and support are putting the pressure on ggLeap management and owners, 4 devs x 240,000 PP = 960,000 PP

Trust me… we feel it more than anyone can imagine.

All of us at ggCircuit are not software people first, we are game center people first.  So I know firsthand the pressure on staff and customers at a store.  My stores are down at the same time everyone else is down.

So I have two crushing problems at this point on me personally that might seem to be in contrast, but actually push us toward the same place in the end:

As a store owner I demand 100% up time
As the CEO of a software company I want to provide 100% up time

So as my software CEO self, I would say to my owner self: “Hey, we are doing the best we can building a cutting edge solution and we are in early access!”

Then as my owner self, I would say to my software CEO self: “This is the lifeblood of my business, every second offline costs me money!”

Here is the bright side of the entire process.  We are as laser focused as ever on our path to unify centers and gamers through software from all over the world.  We still have never doubted our choice to build a cloud-based system.  We will take the responsibility when necessary for our growing pains and always promise to work around the clock to limit any downtime.

We will be in full release in less than 2 months.  Those running the software with us now truly are PIONEERS in this journey.  That is why we are allowing their contributions as pioneers to apply for future license fees and letting them use it basically free of charge during this early access.

Just know that we will look back on this post in a year when we have tens of thousands of locations and millions of players and the pain won’t seem as bad as it is today.  That is when companies are paying us to reach our customers and all of our stores have extra +$$k in revenue that would not have had on our own.

 

Leave a comment