What are game developers thinking! From the horrible first day of the Diablo III launch, where many players couldn’t use Blizzard Login servers to launch their game, to the buggy and laggy launch of Star Wars Galaxies (Sony online entertainment, 2003) or the Long queues that Aion (NCSoft) players had to suffer in the first days and weeks or the horrible design flaws in Fable III (Lionhead Studios) which is supposed to be an established franchise after all, it is becoming clear that most game companies are having quality control and planning issues. What is interesting to know is that no amount of Publisher intervention (yes, publishers are supposed to help with the planning!), preorders (and hence money in the bank), QA team members or prior success stories seem to prevent these issues from happening over and over again. Why, even World of Warcraft ran into the issue of having way too few game servers at launch, blocking many subscribers from logging in or keeping them in long queues, and we all know how that little title turned out.

In this article I will attempt to describe some of the most common reasons why these issues happen at launch, what developers can do to prevent them and to mitigate their effects on the longevity and public acceptance of the game, along with tips for players, who already have invested in the game, and how they can make sure that the games they love get better rather than worse at release.

1. Why are launches and releases so buggy these days?

First I’d like you to know that the rate of buggy game releases hasn’t changed much over the past thirty years. Crashes, Leaked Alphas (Doom, Id), Critical design/implementation mistakes (like the Pause Trick in Mega Man, Capcom ), and non-games (Battlecruiser 3000AD, Take-Two Interactive) have always plagued game releases – what has changed is the large amount of money spent on promoting (read: hyping) games before launch, and hence the large day-1 customer base.

In the old days, when you released a game that crashed in the first hour on a disk, you would immediately be informed within the first few thousand purchases, which might have happened within a few weeks. Global releases were also very uncommon, and games would normally launch in one neighborhood, state or country before going global. This would give developers a lot of time to fix critical issues. Nowadays game launch globally and preorders of millions of copies are not unheard of. The spread of the internets also allows frustrated players and critics to quickly and efficiently word their concerns about bugs. Also, developers don’t have the luxury of a few weeks’ worth of debugging before the majority of players encounter the issues, so games are not necessarily buggier – the first wave of players for AAA games simply has grown from a storm into a tsunami.

 

2. How hard can it be to get this feature right?

As gamers we tend to expect games to get well established features right, simply because it has been done well in an earlier title at least once. For example, game matchmaking in Halo 2 (Bungie) simply worked, so did the Auction houses in WOW or the economy and metagame in E.V.E., or the movement mechanics in Quake 3 Arena, however we forget that none of the other developers have access to the code written for those earlier titles or the underlying designs and formulas, so, realistically, each team will have to start their effort pretty much from scratch. Also, copying a feature directly from other games – especially when the community praises that feature a lot – makes everyone on the development team queasy and the practice is legally suspect. Making games is not getting easier over time, so don’t expect releases to get less buggy for that reason.

Software is also getting more and more complicated. Gamers expect graphics and physics to simply work, without ever thinking about different DirectX and OpenGL versions, different graphic cards and shader compatibilities, physic engine complexities, 3^rd party addons that complicate code and come with their own bugs. Some examples include SpeedtreeRT, Morpheme, Allegorithmic Substance Air, Beast Global Illumination, Third Party Installers and DRM software, etc. If a company decides to use a third party game or graphics engine, they are pretty much at the mercy of the engine developers when it comes to tough bugs. This means that as the number of third-party components grows, games will end up with more and more hard-to-solve bugs.

Complexity in software fosters specialization amongst coders on a team and team sizes grow. This in turn means more management is required and finding a good manager who has a good overview of all the details in the code becomes impossible. At some point the amount of communication necessary to have all coders understand all system interdependencies grows to such an extent that it simply overwhelms the team’s willingness and patience to do so, and things go south from there.

Now, you may argue that a game cannot possibly more complex than say the self-repairing industrial-grade software that modern satellites use, and yet I would argue that that is not the case. The number of languages, tools, and third party SDKs used in a large game are immense. E.g. I worked on a game where we used: C++, C#, Java, VB, Lua, Python as our main programming language, and a host of other languages whenever they came in handy. The number of lines of code written for a major game release is around 1~5 million lines and sometimes more than that (not even considering the code going into third-party engine tools and SDKs) – but the complexity does not only come from size, it is a result of the variation in the systems and their inter-dependencies. Will we ever do away with this problem, I think not – because as coders get better and better at handling these complexities, the array of choices to accomplish anything in code also expands, and with choice comes variation, and that ends in more complexity. Complex yet perfect code will simply never exist.

 

3. Do the “devs” even care?

When players talk about a game, they are always talking about the “devs” doing this wrong or breaking that feature. Most AAA titles have between 50 and 500 Developers organized in different teams, each under a host of managers – there is no “dev”, but rather a disparate collection of minds! As the number of managers and developers grows, so do the opinions on what should be done each step of the way. In my experience, it is always the loudest devs, and the managers with the best connections to the top who end up getting final say, and those two groups are not necessarily the ones that have the interest of the game or players at heart. Many a feature that was cut and missed by players, was also missed dearly by some of the developers. As a matter of fact, the features that you believe are bugged out, make the devs who actually care cringe every time they see them, and yet they are not allowed by management to work on them because other features or bugs are prioritized.

In the end, a game company is a group of people with different tastes and agendas and that will never change. Power-hungry people will always rise to management, and people who truly care will always be in the weaker position – put there because they care! Hence, any feature that can be justified by managers to managers will always gain top priority, whether it is important or not, so don’t expect any changes in that department either.

The Majority of game developers employed at AAA Studios are young, open to taking risks, and have little initial investment in the companies they work for (e.g. no stock, no partnership, no clear career path, no guaranteed bonuses, etc.). Couple this with the high turnover and burnout rate in the industry – a majority of game developers leave the industry within the duration of their first three projects – hence few developers in large companies get the chance to identify with, and be accountable for the games they produce. This means that when the manager asks them to ship a defective product, they nod, feel bad about the lost chances, and walk over to the next company or project.

Game development, especially towards the end of large projects, is notoriously messy, depressing and hardly rewarding. If the game is not well-received by the audience, and few games are these days, that makes to process so much more frustrating – and younger employees are not good at dealing with this kind of situation, thus productivity and morale drops sharply towards the end of the development cycle, when the need for creativity, energy and commitment is at its peak.

 

4. Are QA Guys Blind?

QA departments are rather large these days and suffer from the same politics issues that any large team suffers. There are also many third party companies that offer QA with varying degrees of conflict of interest. QA is where most people who say they are in “games” work, however due to many reasons QA in games has never been and probably will never be on par with QA in other industries. When a Boeing QA tester tests a flight system, he has been schooled and trained for what he does. He also does his job knowing that his decisions and performance can affect the life or death of many people. QA in the entertainment industry is not held to such standards, and accordingly is not given enough resources and the power to make decisions.

In most companies, QA is treated as a help to devs, playing the game a lot so that the devs themselves don’t have to, and given almost no power to choose which bugs are important. As a matter of fact that isn’t even decided by devs – it is the managers who “know” best. I have talked to many QA people who were sick of reporting the same bug over and over and over and then seeing it pop up in Beta tests, on the release, and caught by players. Sounds crazy, but this happens regularly. True, the bug was reported, but the people who are supposed to care about that bug, ultimately have no say whatsoever in the priority it is given. It is not uncommon for devs and managers to mass-change the status of unfixed bugs as fixed, unfixable or not-important, once the number of bugs reaches the ten thousands, and without the person reporting the bug having ownership of it, there is no way of preventing this either.

Will QA ever be promoted from being a grunt and servant to the force that watches over developers? I doubt it, since humans (developers) are notoriously protective of the social positions they hold. Unfortunately, the high turnover rate in the game industry also ensures that lessons learned from one project will rarely be carried over to the next, hence only a power from above can enforce these lessons on developers, and many large-budget games will need to fail miserably before publishers start pushing developers to give QA teams more structure, responsibility and power.

 

5. Bigger, Better, and Cheaper!

Many players find the recently established practice of releasing/selling day-1 DLCs or selling power-ups for real money abhorrent and expect games to be released completed and with all bells and whistles working! After all, isn’t that what we expect from all other industries? Most forget that game companies are in a far worse situation economically than other businesses.

A major title costs around 20-50 Million Dollars to produce and another 10-20 million to market and publish. After stripping out all the discounts to stores, Publishing costs, marketing costs, etc. that kind of game needs to sell at least 1.5 million copies to recoup the initial costs. Then it needs to pay its shareholders and Top management at the publishing company and the developer itself, pay for maintaining and patching the game and for servers (with MMOs and multiplayer games).

The price of major game releases has gone from 40 dollars up to 50 during the past 10 years, while at the same time the price of transportation e.g. and food has risen by a factor of 4 or more all over the world, so the margins have shrunk even further. Now, sales of 1-2 million copies seem like a small number compared to other large industries, but don’t forget that most of those customers have a low budget for purchasing games nowadays, and the number of outlets competing for their pockets is huge. My personal budget is around 10 games / year, and I think long and read many reviews before purchasing any game these days.

It is true that the 50 or so bucks you pay for a game is a lot, and yet the costs of creating assets is rather high. For example, consider that cool racing game you enjoy so much: besides the company overhead for office space, top management, lawyers, cleaning, etc. the cost of creating a highly detailed 3D model and texturing it with specular, normal, detail, color and damage maps is the cost of paying a concept artist for about two days, a modeller for about seven days and a texture artist for another seven (if you are faster at modeling, don’t forget that there will be iterations and changes the leads request). Not even considering all the other professions (who wrote the code that exports that file from .max into the proprietary engine format?). Now, 2 + 7 + 7 = 16 which is about half a month. In the US that is about 2000 dollars of pay + the taxes and healthcare and benefits etc. that the company needs to pay, so about 3000 dollars.

There you have it: about 200 people who bought the game are paying just for that one car model. Now imagine games that have hundreds of customizable cars, tens of tracks, a GUI that makes you want to lick it, and engine code that renders everything beautifully – do you still think buying a copy of the game gives you the right to a couple of free DLCs?

So what do CEOs of large game development companies do to stay in business, and get rich besides that despite these horrible costs? They milk you as much as they can. They use cheap out of college talent wherever they can, and they outsource as much as they can to China, and that works for a while, until the investors who put their money into games find out that a company like that is simply not viable. Then they reset the button, and everyone starts in a new company. Sounds depressing? Hey, it’s just business.

 

6. Can the devs help it?

How can developers deal with this bug(gered) out situation? There are possibilities, even though most companies have been reluctant to try them:

1. Reduce the scope of games – focus your spending on replayability and design rather than art assets and visual fidelity.

2. Join together in consortiums and share/swap assets, code and artwork. There is a lot of reusable material sitting on those Zip Drives in every Game company. You would be amazed at how different a monster can look with just a little bit of tweaking and recoloring.

3. Analyze a larger number and variety of prototypes before going into production. Spend more time in project selection and pre-production, and get third parties and the audience involved in reviews at this stage before committing to full production. Also, try to have a transparent prototype selection process -this will ensure that only the best ideas get through to production, and not just ideas coming from the “top”. Put your prototypes out there, and allow gamers to give you feedback before you ever commit to any one idea.

4. Some gamers will hate me for this but I have to say it: Larger companies need to learn from the successes of the quirkier indie games. They need to go “indie” – in the sense that:

- they have smaller teams and more varied products

- each team keeps in contact with their fan-base and tries to establish its own niche

- each team has more creative freedom in what they do, all while sharing resources (such as game engines, QA, development tools, knowledge and expertise, etc.) with other teams.

- each team also needs to be accountable for the games they produce, maybe even being treated as a profit center and done away with if it does not perform as expected. This will give developers incentives in sharing the profit from their game, and also accountability:  they know they will summarily be fired if their game tanks (that includes managers whose teams fail – this system should not carry over team members who have failed to other teams or projects in the company.)

5. Invest in, and research tools that allow for procedural generation. This includes

- texture generation

- 3D model generation from photos

- Playfield Generation and automatic content placement

- Smart NPC AI that allows for varied behavior depending on its awareness of the world

- Automatic Mission Generation

- Automatic Story Generation

6. Allow for user-generated content, gameplay and feedback as much as possible. This includes allowing users to create and share:

- resources (create a working market and economy)

- defining large-scale events thorough player politics, territory control and espionage

- missions (using existing game resources with the player’s own twist on the plot)

- levels/playfields

- NPCs, ships, building, clothing items (players can model these based on developer specs, and if they gain community approval within a voting system) they stay in the game

- Metagaming where players participate in creating new stories

- Extensive Betas that last as long as necessary and allow the player community to do most the QA and input their ideas on where the game should go.

There are obviously more ways to keep production costs low, and we will discuss them in forthcoming articles. Now I would like to take a moment and suggest ways in which players can help studios not release half-baked and buggy games.

 

7. What can this little gamer do?

- Read user reviews before any purchase. Don’t trust critics on game magazines or industry website reviews. Most journalists on major game review sites and magazines are wooed and paid for by game company marketing departments. There are plenty of games that get 80+ % critic scores on metacritic.com and have user scores lower than 40% based on tens of thousands of user votes (see Call of Duty: Modern Warfare II, which has a 86% critic score and a 37% user score).

- Do not buy games from companies that have promised the sky and delivered dirt on prior releases. You cannot be a vegetarian and have your steak – in the same way, you cannot complain about bad practices in game companies and still play their games. In no other industry, outside the entertainment and possibly the diet pill industry, do consumers fall for the same lies over and over again.

- Take part in betas. Put time and effort into finding bugs for the games that you like to see released. Be active, and join the conversation on the forums. Use the developer’s vanity – they do check the forums all the time for comments about their “baby” – and let your concerns be heard. Also, avoid fanboyism, trolling, being baited or hate messages – those don’t work. Instead try to add as much analysis and intelligence to your comments as possible. Developers read your comments, and if they agree with you, and there are numerous threads on the issue, they can print those comments, take them to their managers and say: “See! This is what players want. Now let me give them what they want for Pete’s Sake!” This does happen, and you can be the proud commenter who allowed it to happen.

- If you have your own webspace, create in-depth reviews of games that you have played extensively. As a developer I am a lot more likely to trust reviews from well-known players of the MMOs I play, rather than reviews from failed journalists who are paid by the word to play a game 10 minutes, copy-paste forum comments, and make a living off their drivel on “professional” review sites. Once you are educated about good and bad games, it is your responsibility to educate the collective.

- Get to know the work of individual game designers and developers. Generally, it is not a “company” that F’s up a game but rather a few bad apples on the development or publishing team. Track their careers, and see if you can find a common thread – believe me, a bad designer does leave a track of failed projects and bad games, especially when he/she is in a lead position. The same goes for lead coders and CEO/CTOs. Obviously I am not talking to the average gamer here, but journalists, and those who enjoy this kind of thing will delight in digging deeper and sharing their finding s with us. If you can expose the bad apples, they will have a hard time finding employment in lead position again and spoiling other games for us.

- Avoid comments on forums or websites that say: “This game is Shite!”. This only dilutes the discussion. Share your reasons in a logical and thought through way. This way your word will carry more weight with the discerning public. If a game is bad, and you played it and know how bad it is, the public deserves to know about it and you owe it to them to know why they should avoid it. If you trust other reviews before buying a game, the public needs to be able to trust you. So avoid fanboyism, lies, hatred and weak arguments, and state facts and reasons.

 

Since there will certainly be more botched game releases and MMO launches, yours truly will be alert, and continue this thread with more lessons learnt from the industry. Have you been through the harrowing experience of a botched launch as a developer or gamer. Let us know what you felt, and what you did.