Lore Lords is coming back…
It has been a few years since my last post but as Lore Lords has been in development for around 35 years, in one form or another, has it really been that long… relatively?
I now have a requirement to learn newer technology; to this end I am going to investigate microservices and scalable deployment – using LLOB as my case study.
After over a year when I have been busy with other things, I find myself with some time available to work on Lore Lords again.
Going through my notes and cloud servers I was able to get going again with my action list.
So strange coming back to something cold after so long, but I’m finding my way around the code and getting to grips with where I left off.
The first action was the completion of the distribution of the game onto the cloud server with control and monitoring through a live database.
I now have the game checking the main server (which is on a different machine) for what games are available and what to do with them, then a state machine ticking over a phase at a time if the game is running or putting the game on hold gracefully if I want to bring down the server.
The good news is I can login to my game server using ‘UbuntuBash’ on my Windows 10 PC and ‘nohup’ a LLOB2 service into the background which stays resident so long as a live game is on the system. As the game is also the web server that both throws up my retro HTML web page form and also runs the HTTP API interface the game service does need to be running even when the game is between phases and turns. With ‘nohup’ I can close my shell without killing the game.
Control and monitoring of the game is done using phpmyadmin at the moment. This shows the phase and turn number as well as current status of the game. I can also change game status in this way (I’m not developing custom management pages yet, no need during this early stage).
It’s nice having the game running and playable, but I still don’t have a gateway developed to allow users to gather and launch a game themselves – at the moment I have to enter their data into the game on creation (or manually through the HTTP API).
The next step will be to have new players be added and editable through the database rather than in the initiating routines in the code. In this way I will be able to add and remove players during a live game, this should make testing with more than just one user (me) possible.
A Long silence…
OK, It’s been a long time between posts. Other than getting commands to work from the master database on the distributed server I have barely done anything on LLOB2 since May.
The last six months have had me 100% busy with client work and I have also started my Masters in Advance Computer Science (Part time). Between these two activities and the resulting US and UK timezones being used up, I am barely getting a few hours sleep most nights and no time to do anything on LLOB2.
On the positive side I am studying High Performance Computing, which is the art of getting massively parallel systems performing tasks – handy for simulations – and Web Development using C#.NET MVC5, the Microsoft way of doing client/server work. The latter may not be so useful for LLOB2 now that it is Linux based, but is very useful for my clients.
One day the client work will slow down – at that point expect posts to be restored to the more normal frequency.
LLOB2
Going SQL
Stephen Mitchell:
“It’s been over a month since my last post, and I have been very busy.
Sadly I have not been very busy with LLOB2. Clients needs have taken priority for the moment, I have had only a few hours to do more with Lore Lords in the last six weeks. However, even in that small window I have taken the opportunity to make some progress.
in 2009 I had been working on an SQL version of LLOB2, I have taken the management side of these data tables and installed it onto a cloud server. As this means I have an SQL system exposed to the internet I have taken the precaution of using an SSL certificate and making the site all HTTPS.
This means, as planned, I am using phpmyadmin to update and tune the management table.
Alongside this secured cloud server I have setup a 2nd server which will allow me to test the method of having multiple game servers communicating with one game management database. Although my design notes are ready for the actual implementation, I have yet to write the changes to the game to use this database to drive the turns/phases and game state.
If I get a few more hours this month I will progress with getting the game to take its commands from the table in the master database.
I still do the initial installation of the game myself, but over time I plan to write a method to allow the game to populate key elements (players etc.) straight from the master database. This will allow a web portal to exist for players to congregate and launch a new game.
Currently the game state is saved and loaded as one file on the local game server. I may switch this to a blob in the master database or even create tables as a way of browsing a snapshot of the game and allowing easier edits.
This is, of course, dependent on my getting some spare time.”
LLOB
The Cloud?
Steve Mitchell:
“Testing using virtual servers on my laptop has been a great development tool, but this is not going to be practical when we start multiplayer test.
Right now I have to port forward on my domestic router to a locked local IP address of an Ubuntu VM running on my Mac laptop. Oddly, I develop using a Windows VM that also runs on my Mac – yes 3 OS’s all running side by side.
My Internet provider has the ‘good sense’ to randomise my IP address on an at least daily basis, probably to try and force me to pay for a static IP.
Although there are port forwarding services and ways I can get my code to identify a shifting IP, having a fixed IP and pointing an actual domain to it would make sense. Also, I am not planning on running my laptop 24/7 stuck to a domestic router.
Time to put the game onto a real server?
This is an issue faced by many developers today, do I plug-in a real server, go Cloud server, lease a hosted real server or perhaps even a hosted virtual server?
The costs on the Cloud look fairly cheap superficially, but it is trendy and the Cloud marketing guys have put in hidden charges that can balloon quite quickly.
Virtual hosted servers are cheap, cheaper than a hosted real server and probably more reliable. They are almost a cloud server, but you don’t upgrade or manage them in the same way.
Having a real server in my office would be great, but the internet upload speeds in Monmouth are nowhere near as good as at a hosted location, so I ruled that out.
Anyway hosted servers provide 24/7 cover, and I don’t want to be standing in a smoking wreck of an office with a fire-extinguisher clutched in my hands or driving in at 4am to recover a downed system personally.
So is it the Cloud or a hosted Virtual server?
Well, scalability suggests the cloud. They have pretty dashboards and API’s to let you control the servers. Even Microsoft’s Azure doesn’t lock you into Windows and actually gives you access to Ubuntu (and most other types of server OS). I’m evaluating Azure (yes, the free evaluation), but it appears more costly and less easy to use than 1&1’s cloud in the medium term.
I have also taken the free month for a minimal cloud package from 1&1 (which turned out to have a £10 connection fee, so, not actually free in my book – marketing people are lovely aren’t they).
The upshot is now I have a single cored minimal cloud server with 30GB SSD and 512 MB of RAM.
That is not a lot of RAM.
On a small server the desktop for a GUI will be a luxury. So, no desktop and thus no graphical user interface. I’m going to need all the RAM I can get my hands on for the game.
This sounds great. However I originally developed a GUI test interface using Tkinter, after all I thought RAM was cheap. I now realise it’s not cheap on the Cloud. So out goes an RDT GUI and in homes SSH (Secure Shell).
OK, I could upgrade my cloud server and just RDT in. This would let me run my exiting GUI test interface directly. Or, I could just admit I should have supported a test interface that would work on a client machine’s web browser in the first place… because I am going to need one now.
On the positive side I do get a free static IP address with the cloud server plus unlimited bandwidth even with their cheapest cloud package.
So it’s the £4.99 / month 1&1 cloud server.
Servers are like IP packets or emails, we don’t care where they go – just so long as we can access them securely, and they work reliably. The fact that I have over a dozen redundant servers all much more powerful than the one that I am renting on the cloud is just a painful fact – that our super slow regional data connections has made all too true. Fancy an eight core, 24 GB rack with 4 Drive RAID – the low height graphics card alone cost much more new than people dump a truck load of old rack mount servers for on Ebay (bidder collect).
In short, the Cloud has won. As everyone is going there, if we want fast connections and unlimited free band width, I guess I am going to have to go there too. I just can’t get anything like the bandwidth in my home city let alone into my actual office.
Now I did shop around. Azure looked cheap at a first glance but was substantially more and for only a 1/4 of a CPU (and what is a 1/4 of a CPU anyway).
I think Azure planned to charge for a host of other things later as well (like bandwidth maybe). The free credit they gave me appears to expire really quickly as well.
There is no such thing as free credit – just backloaded premium charges!
Minimal Ubuntu 14.04 LTS – here we come! Yes, with just SSH access – but really, it is all I should need.
And there is my real problem; currently the test game is a single python program…
OK, yes there are multiple threads etc. but really I just run the one program and everything comes off that, data, web server and processing threads (even the test interface GUI is launched in a daughter thread).
What I really want is a master script, which I can manage from the SSH connection, that runs periodically.
When active it should check the database which contains the control information for what games are to be run and when each one needs to listen, process and ultimately output to the users.
I had developed a prototype for just that in python years ago using an SQL database. It was easily managed with PHPMyAdmin – which was all you really needed, from an ‘admin’ point of view. (I would suggest you never let a regular user on your SQL database with PHPMyAdmin.)
Sadly, after implementing that, I then made the mistake of trying to put the whole LLOB game into an SQL database. You can imagine the pain and misery. I learned then that SQL databases are for pickling your game object or storing your management information and are not for transacting your every variable change.
All the games I want could be run and launched automatically. Each game would have it’s own port and I could even manage a network of machines automatically, or instead just upgrade the cloud resources (rent more cores and RAM).
To swallow a fly I have swallowed a spider…
If this appears to have begun to look more and more complex, you are right.
and so on…
If only I had better internet and some free static IPs available at either my home or office!
To add to this injury, I’m very busy with other projects this month. This new task will need to be done in the few personal minutes I get between family, life and work.
The Cloud it is – fetch me a spider…“
LLOB
Nearly there…
Steve Mitchell:
“I managed to get my main targets achieved for the month of March. I can now submit returns using Form processing techniques. This is done by embedded HTML forms within the auto-generated reports. Currently I support the minimal set of movement, combat orders and troop levels – which is where I started the original game at. To test this better I also added combat and movement reports for all leaders and settlements.
There are still pieces to be done regarding trade, transfers, messages and tuning the economic simulation (calculating taxation and morale effects). As I mentioned in February, the “Specials” function will also need to be done – I’m leaving it to last though as it is not needed for system testing.
I am coming up to my one year target date for a basic playable “retro” version of the game – this means that I need to focus on just the remaining essential features if I want to get Alpha testing of the retro game started on schedule (some time during April). If I get a few clear days over the next few weeks, this should still be possible.
Below is a ‘screen grab’ of Lord Ceolmund of Bernicia from a firefox browser. Notice that combat reports have been added and the form information for the return is embedded within the report. Orders are currently picked from pop-up lists and clarified by entering the region number targeted. I toyed with the idea of allowing commands to change orders and allow transfers between moves, however, I may simplify this to just a list of move commands as in the original game for now to keep it more retro.
You may also spot that Lord Ceolmund has a large fleet but almost no soldiers left after all his fighting. I have yet to restrict fleet positions to ports and water. I will be adding in the ‘depositing’ of the fleet at the last valid coastal location before Alpha testing – as in the original game. I have plenty of these special cases to complete yet.”
LLOB
First Reports
Steve Mitchell:
“The good news is I have made much progress in the early part of February on the planned ‘reports’.
I can flag each of 40 domains to generate a report .html file containing information about the settlements, leaders, provinces, regions and domains in a manner similar to the original game. I am working with just the original game’s LOD (level of detail) for now, later I can make reductions in detail and eventually, one day, a GUI.
One of the more tricky parts of the return was keeping track of leaders as they passed through each region. Although in the original game you were not guaranteed to see other leaders your regions did report the current leader present and the last leader to pass through. Now I have more memory I am creating a set of leaders who have passed through which is reset every ‘turn’. A ‘turn’ being 14 half days or ‘phases’, i.e. a turn is a week in game time. In theory 160 leaders may pass through or be present in any region – hence in the 1980’s I wrote a limited 2 leader buffer which was flushed as the next leader passed through.
The bad news is I have done very little in the last two weeks of February (other than a bit of testing). That testing did reveal three regions which were not in provinces (bad) and a few other typos in the data (e.g. an upper/lower case “I” confusing two settlement references).
It was fun using my test console to send leaders marching across Bernicia and battling each other for the testing of “last and current leader in region” reports. That reminded me that you are not supposed to be able to kill your own aligned leaders… oops. Well, things like that will be address once I have developed the ‘returns’ (the webpage that you complete and submit to make things actually happen rather than just a large .html page of textual data).
below is a tiny clip of just one province (you normally start with four). I would normally hide what the region grid references are, but there was more than one map in the original game and I am going to work on a “randomizer” to distort maps in the future. Really a random map generator will be needed too (it will use British names and have a mixture of water and land, but other than that locations will shift) – but that will be looked at after everything else is done.
I have picked a ‘detailed’ report extract for just once province in the Anglo-Saxon domain of Bernicia, Lord Ceolmund’s main province rather than one of his three underlings.
[This is about 20% of a typical report data and does not include any of the summaries, domain and financials, combat reports or return elements. I should note that the “sixty oar” is a vessel type, I may need to capitalize unit names to avoid confusion. Also, I have tinted and italicized to differentiate it from the body of the post. Don’t forget the detail level will be adjustable, this is the verbose example of just one of the 160 provinces which are divided between the 40 players.]
The Commander of your modest borough Bamburgh, situated in Central Cheviot [1517], relates that the people number two thousand, that food stored totals nineteen thousand five hundred and sixteen units of food and that there are numerous wooden walls protecting. The militia, whose morale is specified as very high, is built up of three hundred and eight peons, one warrior archer, fifty seven churls, sixty three bow men, one hero and three warriors. The local fleet comprises five curraghs, five longboats, eleven transports and three sixty oars.
Lord Ceolmund of Bamburgh, you are presently at home in the town of Bamburgh in the region of Central Cheviot [1517]. Your military, whose morale you have cause to believe is excellent towards you, comprises twenty nine archers, thirty two provincials, one thane and eighty peasants.
North Cheviot [1469] with its pathetic infrastructure and flat wooded environment has a Norseman Heathen people who number one thousand four hundred and ninety six in total. Their level of spirit is noted to be rebellious whilst economic production rates of the region are absent for gold and silver, average for ores, not too good for stone, normal for wood, normal for food, disgracefully low for utensils, normal for fabric, normal for livestock and nowhere for luxury articles.
The highlands woodland region 1468 borders to the west. The rolling woodland region 1420 borders to the north west.
The gentle hills wooded region of West Cheviot [1516] with its better than average road network has an Anglo-Saxon Christ believing populace of one thousand and eight, whose level of spirit is deemed to be zelous. The economic production rates of the region are void for rare metals, average for ores, acceptable for building materials, better than average for timber, typical for crops, normal for tackle, acceptable for leather and woven stuff, not too good for livestock and absent for luxuries.
The hilly moor region 1564 borders to the south west. The hilly moorland region 1515 borders to the west.
The flat land coppiced region of Central Cheviot [1517] with its typical road and river system has an Anglo-Saxon Christ believing people who number one thousand five hundred and twelve, whose level of spirit is specified to be enthusiastic. The resources rates of the region are void for rare metals, poor for metals, quite normal for building materials, quite good for timber, acceptable for food, reasonable for implements, not at all bad for material, more than average for livestock and more than average for luxuries.
Thane Ceolmund is currently here.
South Cheviot [1565] with its improved road and river system and gentle hills thicketed environment has an Anglo-Saxon Christian population who number seven hundred and fifty in total. Their level of spirit is noted to be indifferent whilst economic production rates of the region are nowhere for rare metals, inadequate for metals, below average for building materials, run of the mill for wood, woefully inferior for harvest, poor for tools, woefully inferior for textiles, typical for livestock and void for luxury articles.
The flat land wooded region of East Cheviot [1566] with its abysmal road and river system has an Anglo-Saxon Christian people who number one thousand two hundred and fifty three, whose frame of mind is termed as being typical. The resources rates of the region are void for gold and silver, acceptable for metals, normal for masonry, run of the mill for timber, acceptable for harvest, normal for tackle, not at all bad for cloth, better than average for livestock and void for luxuries.
You will see that I have added neighbouring region reports. These only report a non-owned neighbour once. Players used to use this data to map out what type of land regions were immediately adjacent to their domain, handy if you wanted to avoid marching into the sea when exploring.
Actual scouting reports were more detailed, but you needed a ‘special’ for that. I will be developing ‘cards’ to replace ‘specials’ (more on that in a future post) that you will be able to keep.
I will dedicate my spare time in March to the interactive forms that will let you give commands, that is the final step needed to be able to start doing basic multi-player testing. After that it’ll be fixing issues and tuning the combat and related reports along with the economy (which is broken at the moment). Although not directly to do with gameplay I will then be moving onto ‘player messages’, game ‘turn news’ feeds (we had both on paper before) and finally those ‘special’ cards. ”
January 2016
LLOB update
Steve Mitchell: “I have updated this site to include some of the original cover art from Lore Lords. Later I will be converting the whole original art set into a default ‘theme’. My plan is to allow players to customise and share their own themes later, but the retro theme will be a good starting point for now.”
“On the code side, I’ve continued with testing the integral HTTP server to make sure it is doing what is needed. To this end I have added POST and GET support for the message interface, using CuRL to test the POST interface (handy command line tool) and just any web browser to test the GET interface by pasting command strings into the URL bar. Whilst doing this I have managed to get quite an extensive range of commands and responses supported (though just test responses).”
“As a further experiment with Python’s built in web server I added CGI support for python scripts and file support (for example and actual index.html file rather than python HTTP response). Python CGI scripts are not the same as using HTTP to talk to the game itself, the CGI doesn’t have direct access to the simulation for example. POST and GET work great at talking direct to the simulation, but intensive calls (“hammering”) to file and CGI scripts appear to give the built in server a headache. To deal with this last issue I have installed Apache2 alongside the game to support files (.PHP and .HTML) directly if they are needed, though on port 80.”
“I think the game GET and POST interface will run on port 8080 whilst any bulk files will run on 80 as Apache is a lot more load tolerant it appears – good to find out now. Later they will probably run on different servers.”
“This whole end of the game is critical to get right, and it appears I will continue to work on making the direct HTTP game output simple web widgets, with basic hyper links to control the game at the start, until the retro version is complete. Once that is done I will add an Apache layer for player account control (registering and customising), interface with a database for all games and users (players) on that site and then have links to the game servers from there. Intensive files and links (art / audio) will probably be shared on the general Apache server with only the media location (URLs) stored within the game. Moving .JPG and .WAV or .MP3 files within the game is not as efficient as using links to more powerful servers.”
“Next I have to complete 5 basic report classes with 7 levels of detail each and also need to generate the interactive command links that will go with them (what were the ‘return’ sheets). This will keep me busy.”
December 2015
LLOB update
Steve Mitchell: “I’ve been busy with Lore Lords in the last few weeks. After adding recruitment targets I decided to switch focus to the ‘testing interface’ to be able to start issuing and reading system messages. These messages are how the game simulation is going to react to the human players and AI players. I now have the first command issuing message working (setting the destination for a target leader) and resulting reporting (confirming the change). The test console interface has been added using a windowed display with some basic buttons and text areas, however it will allow me to issue any command as any user and see any reports – very handy when testing. I can now set any one of 160 leaders to any of 3120 destinations and watch the debug data as they fight their way across Britain!”
“I’ve even tested including a built in HTTP server, as eventually external users will communicate with the game via either email or HTTP. Although initially running through ‘localhost’ on port 8080, I will switch to a static IP accessed via a domain name later when we get real users. However there are no web pages (no PHP or HTML files on the server) planned. My initial tests are using HTTP and GET for incoming data and python creating a dynamic return HTTP response for debug. As my standardised message structure currently uses XML (less curly brackets than JSON and more familiar to HTML users), I will add initial support for that in a bidirectional connection. JSON may be added later, as many programmers are becoming more used to this data interface.”
“Connections will be authenticated by username password. Early days here however, just testing combining the HTTP server running in python within the same program as the simulation. However, I won’t complete the HTTP prototype interface until I have a need for multiple human users. I can cope with my resident test interface for now. Nice to see the HTTP proof of concept up and running though.”
November 2015
Lore Lords of Britain development continues
LLOB combat and movement complete
Steve Mitchell: “I have leaders moving and fighting, on land, sea and sieging settlements (they didn’t fly in 890, so no movement by air). As I have more processing power and memory than in the 1980s I find myself thinking about improvements on the original design on a regular basis. I will fix and tweak as I go, but still plan to leave the first version as close to the original as reasonably possible. I should add that some of the features added to Lore Lords in the 1990’s will also be added back in before Beta testing, they were an important part of the game to later players and I would not wish to go backwards. My next task is to manage recruitment targets for leaders and settlements and handle both local trade and inter leader trade. I will probably need to make advances on trade as this was an area where some players struggled in the original games – if they did not have a reliable group of friends playing – as sending commodities was on a trust basis it could be abused. The 2016 version of the game will have a transactional trading system, you will get paid for what you send as it arrives. Though you will be able to steal commodities and carry them off by capturing settlements, so some level of theft will definitely be permitted…”
