Felt good to get this one knocked out:

https://presenta.silogroup.org

That project took like a year of design in between other projects.  I needed a distraction from surro linux for a bit while I wrapped my head around the complexity of bootstrapping compiler builds.

New Year’s Eve is tomorrow.  The lady and I are going to a fancy place to watch the ball drop.

Speaking of balls dropping, the gym visits have got my test levels up.  Kinda digging the new me.  Or is it the old me?

I can’t tell.

Oh, I got one of those Amazon Echo Dot virtual assistant things for Christmas.  I’ve got it going up in the house and it’s pretty cool how it changes the atmosphere in the house.  It can do a ton of things.

I really want to make it just sound an alarm if one of my servers goes down.

IRCThulu is today.

There’s going to be alot of fallout.

Expected Kline

The freenode network is mostly impacted by this.  These are not people who negotiate.  I’m expecting that.

First, the Freenode network is going to kline all of my resources from the network not realizing that there are 3rd parties running the runners who do the logging and that it’s not up to me which channels they log.

Second, this will be the demarc for active development for IRCThulu allowing some time for it to burn in with changes here and there.

SURRO

From there I’ll be able to focus solely on SURRO while enacting the much needed sanctions that IRCThulu provides.

We’ll need to get that mailing list set up.

GitLab has become too resource intensive.  The server just wants too many resources for what it’s actually doing.  I’ll be looking at transitioning our repos to a more efficient system.

As for IRC presence, SURRO will still have one, and I will still be on the freenode network, I’ll just be doing alot of host-hopping in between repeated klines.  It’s unfortunate that this level of upset needs to be created to get such serious issues addressed properly.

The answer is, ultimately, to turn the list of bans in #surro-linux into a kline entry list, and, this is a permanent new age of accountability in these communities.

I’ve decided to give IRCTHULU a new dedicated home.

http://ircthulu.silogroup.org/2017/12/23/new-dedicated-site-for-ircthulu/

It’s painfully clear the types of controversies IRCThulu will cause, and I wanted to document my adventure there in its own dedicated space so as not to clutter my already cluttered blog.

This blog was intended as a scratch board or hub for my projects, so, I can certainly cover things I’m looking at with it, but, it would not be useful to have all content from all projects in one blog.

This is one of the reasons the surro linux blog was moved to linux.silogroup.org.

After a week or so of tweaking the backend to get it stable and initial buidout of the decoupled API that serves the aggregated content — it’s safe to say that IRCTHULU is production ready.

Not only that, after some research it’s also pretty clear that we’re not just talking about a log aggregator — it’s the world’s largest IRC Log Collection Engine currently.

One thing I’m running into is delay in getting the data from the database due to the size of the tables the logs are stored in.

I may want to set up some jobs that create cached tables periodically to speed that up, but, every piece of action you take to do things like that take away from the realtime nature of the data itself.  I’ll have to give it some thought.

This content is password protected. To view it please enter your password below:

Password:

I reworked the consumer in data and have titled it synapse.

Performance Notes

Based on previous tests and the current behaviour of ircthulu’s tenta and data backend, I think we will not even need to consider scaling until we’re at about the 2000 simultaneous channels mark.

It’s there if we need it.  I can cluster and autoscale it easily.  Once we do that I seriously doubt if we could not get almost every major IRC channel on every major network logging simultaneously without significant data costs.

Stability

The Tenta layer is decentralized and distributed.  The MQ is clustered.  The DB is clustered.  The synapse between them is rather butch in the multithreaded manner in which it is processing.  I’m overall pretty confident in its ability to store the content without intervention at this point and am hardening the way in which it is bundled for distribution.

API and UI is next

From here it’s the API and the UI to actually serve the content, and just making sure the feeds are still coming in.  Some are already active.

Participants

If you have a persistent connection and would like to run the tenta client as a systemd background service please reach out at punches.chris@gmail.com with details about the channel you would like to host.

 

I’m pleased to announce the Tenta client came together quite well, marking the completion (and activation) of the data backend for IRCTHULU.

For those of you who haven’t been following this project, this is how IRCTHULU works:

Tenta is a pair of services comprised of the runner client, tentacle and producer service, called nerve. They run in the background of the systems of the people who collect the data, and are installable as an RPM.  The “transporter service” called nerve pumps the messages captured by tentacle into a message queue I currently own.  Tentacle is configured by the runner and can point to any channel they wish.  I can make recommendations about what channel but I can not decide ultimately.

From there a multi-threaded service called synapse in the storage layer empties out the MQ, transforms, and pushes the messages into a database.  From there, an API has readonly access to the databases to serve information relevant to supply a user interface on a remote site.  The MQ/DB/API combination is a layer called Data.

The UI will be called Presenta.

So that’s Tenta, Data, Presenta.

What does performance look like?

The total throughput of the system was last measured at 30,000 messages per second, allowing very many channels to be logged.

What kind of Data does it record?

Here’s a data sample:

{"type": "pubmsg", "nick": "Mihaita", "channel": "#freenode", "message": "Phanes @robot?", "timestamp": "2017-12-14 03:27:36.762231", "host": "166.ip-54-37-156.eu", "ident": "~SirNeo"}
{"type": "pubmsg", "nick": "Phanes", "channel": "#freenode", "message": "SirNeo-, yeah its part of a project called ircthulu, its all google indexed eventually", "timestamp": "2017-12-14 03:28:17.214279", "host": "surro/founder/phanes", "ident": "~Phanes"}
{"type": "pubmsg", "nick": "SirNeo-", "channel": "#freenode", "message": "i will google it about latter", "timestamp": "2017-12-14 03:29:09.333669", "host": "166.ip-54-37-156.eu", "ident": "~SirNeo"}
{"type": "pubmsg", "nick": "Phanes", "channel": "#freenode", "message": "SirNeo-, its not public yet", "timestamp": "2017-12-14 03:29:16.679930", "host": "surro/founder/phanes", "ident": "~Phanes"}
{"type": "join", "nick": "[skyline__O__]", "channel": "#freenode", "message": "", "timestamp": "2017-12-14 03:29:22.442250", "host": "1.180.74.58", "ident": "~Shen"}
{"type": "join", "nick": "blkshp", "channel": "#freenode", "message": "", "timestamp": "2017-12-14 03:29:43.017512", "host": "about/windows/staff/blkshp", "ident": "~blkshp"}
{"type": "join", "nick": "SirNeo-", "channel": "#freenode", "message": "", "timestamp": "2017-12-14 03:29:49.518446", "host": "unaffiliated/sirneo", "ident": "~SirNeo"}
{"type": "pubmsg", "nick": "Phanes", "channel": "#freenode", "message": "SirNeo-, it uses a \"guerilla\" pattern for data collection points from disparate, distributed hosts via a background system process installed via rpm, so there's like no way to tell who it is and it doesn't log any events that could be used to detect the bot through the logs, also waits a random amount of time after starting to begin logging", "timestamp": "2017-12-14 03:31:17.932267", "host": "surro/founder/phanes", "ident": "~Phanes"}
{"type": "pubmsg", "nick": "SirNeo-", "channel": "#freenode", "message": "Phanes, how do you know we don't use allready a VPS?", "timestamp": "2017-12-14 03:31:20.409542", "host": "unaffiliated/sirneo", "ident": "~SirNeo"}
{"type": "pubmsg", "nick": "Phanes", "channel": "#freenode", "message": "channel redundancy", "timestamp": "2017-12-14 03:31:52.678278", "host": "surro/founder/phanes", "ident": "~Phanes"}
{"type": "pubmsg", "nick": "Phanes", "channel": "#freenode", "message": "autorecover", "timestamp": "2017-12-14 03:32:05.244805", "host": "surro/founder/phanes", "ident": "~Phanes"}
{"type": "pubmsg", "nick": "Phanes", "channel": "#freenode", "message": "bug tested for \"gotchas\" to kill the bots", "timestamp": "2017-12-14 03:32:21.725860", "host": "surro/founder/phanes", "ident": "~Phanes"}
{"type": "pubmsg", "nick": "Phanes", "channel": "#freenode", "message": "the metadata separation is the cool part though", "timestamp": "2017-12-14 03:32:53.327834", "host": "surro/founder/phanes", "ident": "~Phanes"}
{"type": "join", "nick": "skyline__O__", "channel": "#freenode", "message": "", "timestamp": "2017-12-14 03:33:03.015222", "host": "1.180.74.58", "ident": "~Shen"}


It’s recording the message type, although some message types are restricted (like some CTCPs or DCC) from logging, the channel the message occured on, the message itself if there is one, timestamp of the message (server), the host of the user submitting the message (or cloak if they have one), and the ident value associated with that host at the time of the message submission.  I will also tonight be adding in an even handler for nick changes to capture that metadata for nick changes.

That’s oddly specific.  What purpose do these datapoints serve?

Great question.  This will allow the UI to serve as a search engine of sorts, allowing you to browse logs, and, identify sock puppets, or trends, or even perform statistical research or machine learning algorithms upon for detecting anomalies.  I’ll explain one example use case:

A user FrankG likes to use sock puppets to harass people online.  He connects with FrankG!dirtyfrank@224.223.221.33 normally.  You suspect that he has goofed at some point and used the same ident on his irc client or host at some point breaking OPSEC.  This would easily be determined by simply looking at a log that captured frank talking by searching for username FrankG or going to a known channel and time you knew he was speaking.  From there you can click on his ident, dirtyfrank and see all the users captured who used that ident.  Or you can click on his host 224.223.221.33 and see all the users ever having connected and posted a message in a logged channel from that host.  Or you can click on his username, FrankG and see all the hosts and idents that have ever been used by the username and begin doing deepe analytics.  Obviously, the larger the dataset, the more effective it is.

This is in addition to a more important community need being filled:  IRC is a lost trove of knowledge for people looking to increase their knowledge in a technical area — these channels are like living textbooks for people solving problems, particularly on open source systems.  By providing this resource for as many channels as possible we are opening up those channels to google search indexing, allowing someone looking for a problem a greater likelihood of finding their answer when googling.  Cool amirite?

What does the legal model look like?

Great question.

I own a database cluster that stores the feeds and the UI, which are both on different domains and can be decoupled at any time and even given to someone else.  Both the database layer and the UI publish only the feeds that are given to it and all of the feeds that are given to it.  The feeds are public domain.

The runners are, of course, expected to comply with Terms of Use and EULA for any networks they connect to and I trust them not to create liabilities for themselves with it.  Ultimately this is a discussion between the runners and the network owners.

In terms of IP, I’m keeping it closed source, for now, but obviously if it serves my interests I can make it open source and release under AGPL.  My only concern is that if I do that it will be abused and create many spinoffs that I can not control.  And, obviously, if I have to transfer ownership of the DB or UI layers I’ll lose any ability to control content and will have to release the source.

What’s left to do before it goes public?

I’ve got some adjustments to make in the transformation layer of the synapse component to really polish it out and then it’s the API, UI, and SEO.

How do channels “opt out”?

Some channels or networks may desire not to be logged and this is certainly understandable.

They can simply email me at punches.chris@gmail.com and request it, stating why.

While a reasonable justification is required, the overwhelming majority of channels will be approved with no issue; some will be expected to make adjustments to qualify and some in particular should view those adjustments as sacrifices.

IRCThulu, ultimately, is a tentacle monster by design, it’s the size of a building, and it’s destroying reality on IRC.  Well, not all of reality, just a particular kind.

Cross-compilers are tricky business for this kind of operation.

At this point I’m still picking away at various issues surrounding binutils/gcc/glibc required to do the rest of the build out, but Foster is coming together.

Unfortunately, a community need that I’ve been putting off deserves immediate attention until it’s complete, and ircthulu is one of the components for that initiative, so, I’ll be devoting some time to that — and hopefully one of the greybeards will use that time to unkink the cross-compiler generation in Foster.

Ircthulu is such a large project that I don’t expect it to end soon and it’ll probably be a little controversial for a bit.

As mentioned previously, a new feature was needed to be added to Examplar to allow much greater simplicity in the Foster build to make its content more distributable and I could definitely see that affecting usage due to the removal of the need to hardcode paths in the various files.

So, we’ve added it.  Instructions are in the repo.

The source code for the 1.6b version is here.

The source code for Foster is here.

I believe we will probably have much slower updates from here.

Right off the bat I noticed that there are some features that will make the foster builds quite a bit easier.

The most prominent feature I’d like to see added next will be a config file that sets any enviornment variables the user would like any executables that get called to have in their environment.

Another is per-Task STDOUT/STDERR logging as it’s currently up to the user to implement this in their scripts.

Thoughts for future versions.  In our case we’ll need some variables set across several tasks or heals that will be executed, so, I’ll be sourcing a common file that sets these.

Another is a switch for setting the current working directory for execution context to prevent having to hardcode system-dependent paths all over the place in the units.  I will make sure that gets put in place next.

So you’ve got Examplar compiled and you’re ready to start automating the world.

If you’re thinking “how do I configure this thing”, this article is for you.

Definitions

Units

A Unit is an automation definition, written in JSON in a UNIT FILE.  Deeper into Examplar’s internals, Units and Tasks have slightly different functions, but for the purposes of users, the terms can be used interchangeably.  A Task is a task to be performed in a Plan, and a Unit is its definition.  A Unit is a JSON object that has:

• A name, which is an identifier for the Unit used by people.
• A target, which is the path to the automation script performing the work.  This provides a clean linear path for huge chains of scripts to be executed in order and tracked on return for additional logic in chaining.
• A rectifier, which is the path to the automation script to be executed if the target call fails.
• A rectify attribute, which tells Examplar whether or not to execute the rectifier in the case of failure when executing the target.
• An active attribute,which tells Examplar whether or not the Unit can be used in a Plan.  This gives Unit developers a way to tell Plan developers not to use the Unit.
• A required attribute which tells Examplar whether or not the Plan can continue if the Unit fails.  If the rectify attribute is set to true, this attribute is checked after a rectifier failure.  If not, this is checked after target failure.  In either case, if the rectifier or target do not return successfully, Examplar will halt the execution of the Plan if this is turned on for the unit being executed.  Otherwise it simply moves to the next Unit being executed.

Tasks

A Task is an action item in a Plan, just like in real life.  In the context of Examplar, a Task is a Unit that has been loaded and incorporated into a Plan in an actionable state.  Inactive Units can not be loaded into a Plan and thus can never be a Task.  The primary difference between a Task and a Unit is that a Unit is not actionable — it’s just a definition — while a Task a consumable, actionable automation.

Suite

A Suite is not visible to the user and this is only for informational purposes.  A Suite is a collection of all available Unit definitions loaded from one or more UNIT FILES.  Just as a Unit is the definition for a Task, a Suite is a collection of Units that define the Task components of a Plan.

A Suite is consumed by a Plan during the conversion of Units to Tasks, though this is not visible to the user — it just simply helps to understand the kind of abstraction taking place in the conceptual model of Examplar.

Plan

A Plan is the glue of all the components of Examplar and is deceptively simple.  A Plan loads a Suite for its Task definitions (Units), but the Tasks to actually execute are specified in the PLAN FILE.  The Tasks are executed in the order specified in the PLAN FILE.

FILES

CONFIG FILE

This is the one config file that Examplar uses.  The default path it looks is /etc/Examplar/config.json.

A config file at the time of writing this specifies a single JSON object with 3 attributes:

• The UNIT FILE path or a path to a directory containing unit files.
• The PLAN FILE path.
• The configuration VERSION.

UNIT FILE

The UNIT FILE is a specification of where the Units are defined.  All UNIT FILES in that directory will be amalgamated to generate the Suite.  These types of files must end in `*.units` for their filename.

PLAN FILE

The PLAN FILE is a specification of the order that Tasks are executed, and their dependencies upon each other.  Dependency implementation is a touchy matter that is pending implementation, so, mileage may vary until release.

Configuration VERSION

The configuration version is checked to ensure that the configuration is consumable by that version of Examplar.  This will pave the way for reverse compatibility if the project moves in that direction.

Man have I been busy since my last post.

I got the C++ rewrite of Tenta semi-functional and then hit a roadblock that would require a rewrite of the entire IRC event layer to really interface with NERVE the way I wanted it to.

So, naturally I shelved it because when I hit a roadblock like that staring at it longer won’t fix it.  Network programming sucks because of how optimized it needs to be, and I’m more of a prototyper.  The python version is still far superior but that’s just because the network layer is wrapped up very well in python.

In the meantime I brought Examplar not just to Alpha release status but through it and into Beta.  We’re on version 1.5b now and it’s looking really good.

Good enough to call it stable at this point, but, I want to hold off until it generates a bootable ISO on Milestone 2 for Foster (The SURRO Installer Image).  I have no doubt that as we get into the thralls of building a Linux distro that we’ll want to make some changes or optimizations that will make the first stable release that much better.

Been hitting the gym pretty good.

Putting Tenta on hold for a while to recover from the Examplar frenzy of the last week or so, and in the meantime I’ll pick away at some theme tweaking for the blogs and news feed.  Really need to get that linux blog at silogroup on this CNAME and this one moved over to silogroup though, the SILO/SURRO split was kind of half baked.

It’s been pretty peachy.  Getting my half-mile jog time down and am considering adopting a puppy if the landlord will allow it.

I’m unsure of what direction I want to take the new blog themes.  I really like having a bland appearance on the SURRO blog and absolutely must keep the “Phanes as Solaire screaming into a microphone over a nuclear cloud” theme.  Maybe I’ll just clean it up a bit.

I’m taking a break from projects for a few days and just having fun.  I don’t do that enough.

SURRO Linux is now moving from milestone 1 to milestone 2.

In this phase we will build a bootable ISO where the rest of the build occurs for a lightweight environment from which SURRO will be installed.

Examplar will be the driving tool for these builds and configurations.

-CP

Examplar is now in Beta at version 1.5b.

I’ve removed all hardcoded paths, implemented all relevant commandline switches, usage docs, and more.

At this point the tool is to be used for all SURRO/Foster development.

This version by all manners looks and behaves stable, but, I’m not comfortable releasing as a stable version until it’s built a bootable ISO so that it can have that guarantee behind the release announcement.

Feel free to download it and use it for any unrelated projects.

I expect in a future version that we’ll want to have a switch to disable syslog logging for absolutely barren environments but I do not think that will be necessary at this time.

Since we’ve moved to beta, I will be offering public RPMs and spec files.

Most work from here related to development on Examplar itself will revolve around improving documentation on how to use it as this is a new tool in the Linux space so I expect alot of confusion at first.  It’s deceptively simple.

In the meantime the source can be downloaded here if you’d like to compile it yourself:

https://source.surroindustries.com/Cycle0/Examplar/repository/archive.tar.gz?ref=Examplar-1.5b

It is strongly recommended to package it before installing it due to the frequency with which we’ve been updating, but it really doesn’t need to be “installed”.

Here’s a script that will get you up and running. Make sure you have cmake installed on your system to be able to compile it. Cmake is a great tool for building C/C++ projects and more. You will also need a C++ compiler. I use G++.


mkdir ./Examplar
cd ./Examplar
wget https://source.surroindustries.com/Cycle0/Examplar/repository/archive.tar.gz?ref=Examplar-1.5b -O examplar-1.5b.tar.gz
tar xvf examplar-1.5b.tar.gz
cd Examplar-Examplar-1.5b-*
cmake .
make
./examplar -h


Thanks to all the wonderful people who have provided input on the direction of this project, and I look forward to what we do with this — and I know the SURRO Greybeards have been very patient while I tried to make the time to do this.  I can not say that making the time did not have consequences, but we made it and that’s what matters.

After a couple nights of grinding, we’ve got a major, major improvement from the v1.0a candidate.  Examplar v1.3a is released.

This version has major bug fixes and feature additions so it’s recommended to move to this version from 1.0a.

• This version repairs a rather serious logic bug in the task execution flow.
• There is also the new feature of now being able to split Units across multiple files in the same directory and pointing the config to the directory of .units files, which will give some more flexibility in unit definitions.
• It now logs to syslog.
• Reworked some of the exception layer to give more details to the user during failures.

I’m rather pleased with this version although I would like someone to come and clean up the code a little bit as there are a couple parts of it that are functional but ugly.

You can use whatever you’d like to monitor syslog — in my case I use systemd on my test machine, so I just do:

journalctl -t Examplar -f

And it will hold onto the syslog in realtime while I execute Examplar, giving me realtime logs.  It still dumps to STDOUT and STDERR. This should be relatively universal, but, like I said, use whatever you’d like to monitor syslog during Examplar runs.

At this point all original features are accounted for and it’s ready for some hands-on testing by building out Foster — but I think I’d like to hold this in Alpha before moving to a Beta or stable release.

-CP

We made it to alpha.  This is where the fun starts.

In Alpha status, we’ve got all the functionality of the tool implemented but will still be adding in some polishing as we encounter any issues using it to build Foster.  There will be minor features added while in Alpha status:

• Logging to file
• Loading a directory of UNIT FILES instead of just one UNIT FILE.
• Commandline switch processing.

Examplar is released under AGPL and requires only a C++ compiler and cmake to build, with zero dependencies.

Why Examplar Works

So, this has come up a bit, and I wanted to have something written up somewhere that explains how these parts all fit together.

The Solved Problem

During the build of the initial environment, there are alot of considerations and hurdles that come up depending on what distribution you are on, and what you want built or automated.

Needed a Boostrap With No Runtime

In our case, we’re going to be compiling into a chroot with basically nothing but a compiler, glibc, a kernel, libstdc++ and your fingers.  We needed a tool that could, in a distro-independent manner, start automating first the initial creation of that chroot environment, then do the compilations necessary to populate it with those things, and then still have enough runtime to bootstrap into itself to continue the rest of the setup of that environment in a way that accommodated more and more complex tools being executed which depend on runtimes provided by previous automated tasks that had been performed by the tool — in a chain of successive scripts that have no defined complexity limit.

We also needed a way to run environment checks and audits on one track of scripting and work to perform repairs when those checks failed, or alternative scripts to run when automations failed — and current automation tools that are distro independent require massive runtimes that just are not realistic to build by hand in a chroot (ansible, chef, puppet).

Coding Task diversity and Order for a Distro Build had Unmanageable Complexity

In addition to this, it became clear that the nature of automations needed for building these environments was so diverse that abstracting them into units for planning and quality checks was just a monstrous task and we needed a tool that forced us to abstract tasks into work units before planning what needed to be done.  This would also allow us to version control the plan for the work needing performed itself as opposed to managing that process through very talented humans — and we needed the plan to be decoupled from the tasks needing performed so that the abstraction necessary to create a distribution could be focused on without too much low level on implementation.  We also needed the tool to have some level of recursion possible.

We needed the Plan as well as the Coding Tasks manageable by many Contributors

I realized that most of the guides out there were for single users to build single user systems.  I needed a way to allow a large contributor audience to contribute not just to the tasks that were being coded but to the plan that defined the execution order, and what would be executed, in a way that allowed profiles in the build system.

We needed something with a Flexible Use case

We needed a tool that could be repurposed outside of our initial use case to create a toolchain that was reusable to save time since we have a small crew that could become large if we were to beat the odds and create a successful project.

We wanted to introduce a new tool of this type to the Linux world, and we wanted it to be dead simple.

Due to many of the reasons mentioned above it was important to me at least to provide a tool to the Linux world that did not have such heavy runtime dependencies and domain-specific knowledge required to use, while still being able to accomplish most of the net capabilities these tools provide.  We wanted to provide a tool for the systems-heavy DevOps admin and their development-heavy environments.

We wanted modern configuration style

And JSON was it.

That’s why work started on Examplar.

A Pivotal Moment: The Design Evolution

However, during the design process I realized that I could make this so generic that it could do everything without requiring a complex specialized toolchain as long as it was only providing “workflow” capabilities for already existing tools and resources on the system, allowing the automation to grow with system capability.

This would save the eye-crossing complexity of massive makefiles or cmake configurations, while providing a tool that could bootstrap into itself and continue whenever the work was moving to a new environment (a situation that will happen a few times during the builds from “your current distro” to “your booted SURRO Linux Environment”.

 

* * *

 

How SURRO’s Examplar Works

First, some definitions:

Units

A Unit is an automation definition, written in JSON in a UNIT FILE.  Deeper into Examplar’s internals, Units and Tasks have slightly different functions, but for the purposes of users, the terms can be used interchangeably.  A Task is a task to be performed in a Plan, and a Unit is its definition.  A Unit is a JSON object that has:

• A name, which is an identifier for the Unit used by people.
• A target, which is the path to the automation script performing the work.  This provides a clean linear path for huge chains of scripts to be executed in order and tracked on return for additional logic in chaining.
• A rectifier, which is the path to the automation script to be executed if the target call fails.
• A rectify attribute, which tells Examplar whether or not to execute the rectifier in the case of failure when executing the target.
• An active attribute,which tells Examplar whether or not the Unit can be used in a Plan.  This gives Unit developers a way to tell Plan developers not to use the Unit.
• A required attribute which tells Examplar whether or not the Plan can continue if the Unit fails.  If the rectify attribute is set to true, this attribute is checked after a rectifier failure.  If not, this is checked after target failure.  In either case, if the rectifier or target do not return successfully, Examplar will halt the execution of the Plan if this is turned on for the unit being executed.  Otherwise it simply moves to the next Unit being executed.

Task

A Task is an action item in a Plan, just like in real life.  In the context of Examplar, a Task is a Unit that has been loaded and incorporated into a Plan in an actionable state.  Inactive Units can not be loaded into a Plan and thus can never be a Task.  The primary difference between a Task and a Unit is that a Unit is not actionable — it’s just a definition — while a Task a consumable, actionable automation.

Suite

A Suite is not visible to the user and this is only for informational purposes.  A Suite is a collection of all available Unit definitions loaded from one or more UNIT FILES.  Just as a Unit is the definition for a Task, a Suite is a collection of Units that define the Task components of a Plan.

A Suite is consumed by a Plan during the conversion of Units to Tasks, though this is not visible to the user — it just simply helps to understand the kind of abstraction taking place in the conceptual model of Examplar.

Plan

A Plan is the glue of all the components of Examplar and is deceptively simple.  A Plan loads a Suite for its Task definitions (Units), but the Tasks to actually execute are specified in the PLAN FILE.  The Tasks are executed in the order specified in the PLAN FILE.

FILES

CONFIG FILE

A config file at the time of writing this specifies a single JSON object with 3 attributes:

• The UNIT FILE path.
• The PLAN FILE path.
• The configuration VERSION.

UNIT FILE

The UNIT FILE is a specification of where the Units are defined.  In future versions of Examplar this will actually be a directory and all UNIT FILES in that directory will be amalgamated to generate the Suite.  Currently this is only one file.

PLAN FILE

The PLAN FILE is a specification of the order that Tasks are executed, and their dependencies upon each other.  Dependency implementation is a touchy matter that is pending implementation, so, mileage may vary until release.

Configuration VERSION

The configuration version is checked to ensure that the configuration is consumable by that version of Examplar.  This will pave the way for reverse compatibility if the project moves in that direction.

 

* * *

 

What Examplar Looks Like

1. Write a script that does a thing, or checks a thing, or builds a thing, even calls a thing that does a thing.
2. Write a script that fixes conditions that you expect to prevent whatever you called previously from completing successfully.
3. Create a UNIT FILE defining those two scripts.
4. Create a PLAN FILE referencing the UNIT you created.
5. Point your configuration file at the UNIT FILE and PLAN FILE.
6. Run Examplar.  Eat Waffles if Necessary.

Experienced scripters will notice two things right off the bat.  For (2) you cant know all of the conditions, and if you could you can’t have a way to know which condition occurred.  This is accounted for in the design!

Write checks for each of your conditions as Unit targets.  Write heals for those conditions as Unit Rectifiers in their corresponding Unit.  Enable the Unit Rectify pattern.

Only you can predict the things that will go wrong, so you’ll want to approach it iteratively — which where previously was not possible is now gaurdrailed in a system that forced you to approach it as the creation of “code units” that “do a thing”, adding on as you encounter new ones needed.  This tool not only is possible to use in Agile environments — but it facilitates the Agile workflow in coding environments — and the fact that is has ZERO dependencies means it can run almost anywhere, including Windows systems in future versions, or embedded systems, or inside of a cloud container.

Can your current tool do that with this kind of flexibility or are you muscling Agile into your workflow by team willpower?

I am pleased to announce that Examplar subprocess execution has been reworked and the product now fully supports curses dialogs, ssh clients, and much more — giving you full I/O with any processes that are executed by Examplar.

There is one more feature to implement before the product can hit an alpha release, which is dependency management during the loading/execution of units.  This is mostly just managing the data model which is pretty fleshed out by this point due to the decision to bind design to source.

So excited about this.

Hey all.

Well, it’s certainly been a bit since the last update, but, I have diagrammed the execution flow for the Task class, finally, and fleshed out a (mostly) working implementation of the subprocess execution class consumed during that process.

Note:  Green is cheese for the mice.

Note 2: Not really.  Those are user-configurable items in the datastructure.

Decisions

• At this point implementation is almost complete.  I’ve gone ahead and made an executive decision to include the design documents with the source in the Examplar source repository.
• Any pull requests that do not reflect their changes in the design (where applicable) will be summarily rejected.
• I have removed output checks because it’s unnecessary and complex.  If users desire that they’ll need to implement it in the code that Examplar calls.  It is designed to accommodate that, so there is no loss of functionality there.

TODO

• There is a minor issue with I/O redirection that I’m working on (select/poll madness needed there).
• It doesn’t support curses dialogs very well at all yet, I assume there’s additional communication going on regarding cursor position, refresh, and window size that needs to be accounted for.  I will not consider the product releaseable until curses dialogs are fully supported.  This also applies to interactive utilities like an ssh client.  This was actually partially working until an issue with exit code of child process capture was repaired that ended up breaking it in the process.  Everything comes back to the same class for subprocess execution.
• Dependency check implementation.

Summary of Current State

Examplar is actually a functional product at this point and can be used for simple styles of scripting (i.e. tools that have output but dont require input to STDIN) but not for interactive processes.  The difference between a full, robust software suite and current state is a matter of fixing I/O redirection in a single class, which is in progress.

That should be relatively soon, but in the meantime you should be able to use it to build with your favorite python scripts, bash scripts, ruby scripts, C tools, or whatever and even begin building out things with it as long as those things don’t require keyboard entry from the user.

This is, of course, at the time of writing this, so for posterity’s sake the current source is here.

The repos are kind of the wild west until first release.

Musings

As I’m working through the test scenarios and use cases it’s occured to me how badly I needed this tool in some environments I’ve worked in and some builds I’ve done in the past — yet it’s so simple when you zoom out.  I have a feeling this has the potential for a larger impact than I initially intended, in terms of the quality of the automations that will be produced in this as opposed to just scripting or programming something out without rails in place.

All in all, it’s coming together to meet its originally intended purpose:  A scalable complex (or simple) automation or testing framework.  It could even be used for orchestration or configuration management.

I do have to laugh at the fact that the last 5% of the code took 90% of the project time.

Day 3

My home office has been basically transformed into a war room.

As I was planning it and building it out, it occurred to me that the last two years of my life were hallmarked by perpetually not giving myself enough time to plan for my personal and nonprofit projects.  I’ve been trying to do too many things at a time and not completing any of them.

Enough.

I will complete every project I ever started.  I will complete them well.  I will complete them in a planned fashion.  That is all.

C++ Refresh -> Tenta Rewrite -> Examplar Prep

As I’m refreshing on C++ syntax and design patterns for the Tenta rewrite, one thing is clear:

C++ development is very, very slow.

Examplar, also written in C++, is intended to be done after completion of the tenta client as I found that I had some gaps in my knowledge of the language necessary to complete it, and I did not use a sane design process around either tool due to lack of time during the early stages of both projects.

It’ll be worth going back through the motions though.  I want Examplar and Tenta to both be a single binary, a single config, and in the case of Tenta, a systemd unit target packaged up nicely and only relying on the C++ runtime.

With Tenta complete, I will rewrite Nerve, which will be substantially easier to rewrite, and then I will complete Examplar.

Examplar will be branded independently from SURRO Linux but will be a driving component during the early milestones.

Apologies to the SURRO Linux team for taking so long with this, I’ve been up to my ears in work this year.  Documentation is in draft.

I’ve got so much stuff tah-do..

• New email rules.  I’ve got 36,102 unread emails, 36 unused and outdated email folders whose contents need reorganized.
• Chef setup for all of SURRO and SILO.
• Clean.  Clean.  Clean.  Windex cleans everything.  Except souls.
• Run, Run Run.  Gym every other day.
• Hit the weights.  Lower weight, higher reps.
• Reorganize all my cupboards.  This is serious.  I have vitamins, windex, and power tools in the same storage drawer.  Not kidding.
• Continue studies on a potential rewrite of Tenta in C++.
• Re-read some of the POSIX standard.
• Read ‘Beej’s Network Guide’.
• Re-read ‘Unix Network Programming’
• Re-read some ‘K&R’
• Read ‘Design Patterns: Elements of Reusable Object Oriented Software’.
• Start the formal systems design process for Examplar.
• Build the last POC feature for Examplar.
• Prop up a new Jenkins and mock build server.
• Build a shit-ton of new RPMs.
• Harden tenta SDLC.
• Completely redo the silo landing page.
• Redo the SURRO blog theme.

This content is password protected. To view it please enter your password below:

Password:

Hello, boys and girls!  This is a follow-up to an article I wrote a while back about why sysadmins and developers use packages to distribute software.

On today’s episode of “NO”, we’re going to go over how to create your own RPM repo.  As it turns out, this year I’ve ended up with tons, and tons, and tons of software I want to distribute internally and externally with all these projects and need a clean way of managing it.  I even have a need now for a CICD pipeline, which is a thing all of its own.

This is an often debated topic among some of my circles, where, in many cases, the amount of work in building packages and setting up a repo just frankly outweighs the benefits that packaging software for internal distribution and then distributing from a repo has.

I can certainly sympathize with that, so, I thought I’d list out the steps so that I can keep a record of breadcrumbs along this treacherous minefield of hardships towards:

1. Setting up an RPM repository.
2. Configuring client machines to use that repository.
3. Understanding CI/CD benefits with RPM.

I’ve certainly got my work ahead of me on this one as I’ve heard this is just an enormous and time consuming amount of work.

I’ll have to break it down into smaller pieces I think to get it all done this year.

Key:

Items colored in purple are entry-level system administrator tasks that should be second nature to you by the time you really need to have a custom repo.

Items colored in red are specific to the task of setting up an RPM repository or a client to use an RPM repository.

Items colored in black are mostly sarcasm and general assery.

Items colored in yellow are commands that you type.

Items colored white are the output of those commands you should expect.

*I'm extremely colorblind, so, if I got the shades wrong, let me know so that I can tell you how much I really, really....really care.

1.1 Get a Server

Get a machine to host the repo on.  In my case I used a linode VPS.  This is a straight forward process that I can’t explain for you much more than just to say “Pick a VM size, pick a distro, deploy the image, start the server”.  It’s not very hard.

I picked Fedora 26 for my OS which is a kind of testing distro for next-generation RHEL.  The commands will be identical on any redhat-based distribution, such as CentOS, or Fedora, or Oracle, or whatever you are using.

1.2  Point your DNS at it.

This may be a little more advanced for some but I can’t explain how to set up DNS for your environment.  In my case I used repo.silogroup.org by creating an A record pointing to the server IP, and then setting up reverse DNS on the linode.

1.3  Install Apache.

Now remember on Fedora you’re using DNF now which is the next generation of yum.  DNF has legacy symlinks and CLI options for yum but it’s better to get in the habit of typing it now as opposed to later so that you’re able to transition, and, there’s no guarantee it’ll just be there waiting for you in a future version of your distribution.

 sudo dnf -y install httpd

If you’re on CentOS or similar:

sudo yum -y install httpd

1.4 Create some directories.

You’ll need a place to store your packages.

[phanes@repo ~]$ sudo mkdir -p /repos/{CentOS,Fedora,SURRO}
[phanes@repo ~]$ sudo mkdir -p /repos/CentOS/{5,6,7}/Packages
[phanes@repo ~]$ sudo mkdir -p /repos/Fedora/26/Packages
[phanes@repo ~]$ sudo mkdir -p /repos/SURRO/PRE-ALPHA/Packages
[phanes@repo ~]$ tree /repos
/repos
├── CentOS
│   ├── 5
│   │   └── Packages
│   ├── 6
│   │   └── Packages
│   └── 7
│       └── Packages
├── Fedora
│   └── 26
│       └── Packages
└── SURRO
    └── PRE-ALPHA
        └── Packages

13 directories, 0 files

Ok, so, I lied.  In this guide we’re making five repos instead of one.

This is going to be a crazy amount of work, right?  I always do this, I blow my project scope with additional considerations that’ll save me time later, like I’m planning ahead or something.

1.5  Initialize the repository

Repositories, as should be expected, keep some metadata about the packages they’re hosting so that clients can connect and do the awesome things that package managers do.

The grueling process of initializing the repo database is as follows:

[phanes@repo ~]$ sudo createrepo /repos/CentOS/6
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[phanes@repo ~]$ sudo createrepo /repos/CentOS/5
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[phanes@repo ~]$ sudo createrepo /repos/CentOS/7
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[phanes@repo ~]$ sudo createrepo /repos/Fedora/26
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[phanes@repo ~]$ sudo createrepo /repos/SURRO/PRE-ALPHA
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

Man, this is exhausting.  So at this point all of the databases for all of the repos are initialized.  If you’re tired at this point it’s advisable to take a break, because, like, the rest is really, really hard comparatively.

1.6  Open port 80 and 443 in FirewallD.

This is standard for any installation of apache on a redhat-based distribution so you should be used to it if you’re familiar with these systems already (and here is the code if you’re not):

[phanes@repo ~]$ sudo firewall-cmd --permanent --add-port={80,443}/tcp --permanent
success
[phanes@repo ~]$ sudo firewall-cmd --reload
success

1.7  Symlink the Repos to THE Apache DIR

Before you bring up Apache it’ll need something to actually serve, though.

For reasons I won’t get into here, I’ll just say that this step is needed and do not just plop the rpm’s into Apache’s existing document root.

[phanes@repo ~]$ sudo ln -s /repos/CentOS /var/www/html/CentOS
[phanes@repo ~]$ sudo ln -s /repos/Fedora /var/www/html/Fedora
[phanes@repo ~]$ sudo ln -s /repos/SURRO /var/www/html/SURRO
[phanes@repo ~]$ tree /var/www/html
/var/www/html
├── CentOS -> /repos/CentOS
├── Fedora -> /repos/Fedora
└── SURRO -> /repos/SURRO

….symlinks, yay!

1.8  Enable the Apache Service

Like I covered during my disgustingly over-explained systemd pattern series, we will want to enable the Apache service.

I’m only using one mode and set of configs for Apache in this case because I have a dedicated server for a repo.  If you’re lumping your RPM repos in with an already existing Apache you will probably want to do something different (something else, like, not doing that, amirite?).

Just in case:

[phanes@repo ~]$ sudo systemctl enable httpd

1.9  Start the Apache Service

Riveting!  Just in case:

[phanes@repo ~]$ sudo systemctl start httpd

2.0 Adding Packages

At this point you’re probably wondering “Ok, I’ve got an empty repo.  Now how do I get packages to it?”.

This is up to you and largely depends on how you’re creating your RPMs, to be honest.  In my case, I am using a local VM running a Jenkins instance to build some project-specific RPMs but that’s out of scope for this guide.

I’m also set up to use Jenkins to stage the rpm server which hides all the crazy complex work to do that, but for the purpose of demonstration I won’t be doing that:

2.1 Copy the packages to repo

[jenkins@jenkins rpmbuild]# pwd
/var/lib/jenkins/rpmbuild
[jenkins@jenkins rpmbuild]# scp RPMS/noarch/tenta-0.20171015.91-1.fc26.noarch.rpm repo.silogroup.org:/repos/Fedora/26/Packages/
tenta-0.20171015.91-1.fc26.noarch.rpm 100% 13KB 158.2KB/s 00:00

As you can see, while it may be craaaazy complicated, I used scp to move the package to the repo.

I then issued the command needed to update the repo database to make the package available as seen below.  

2.2 Update the repo Database

[phanes@repo 26]$ sudo createrepo --update /repos/Fedora/26
Spawning worker 0 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

You really should have your CI server issue this command if you’re using one.  

And there you have it.  You are now a lean**, mean, packaging-ready machine.*

USING THE REPO / Client Machines

I know what you’re thinking: “He said this would be exhausting and time consuming and expensive but that only took like five minutes and only needed one person.  What gives?”.

Here’s where the work is at:

You now need to configure machines to use your repo if you actually want to use it like a repo.  Scary, amirite?

Fear not, bromance***, I’ve got that mapped out for your project too to traverse the angry seas of package management!

3.0 Configuring Repo on Client Machine

In this case I’m deploying super-seekret mind control technology to a server I don’t want people to use so I’m redacting everything you don’t need to know.

3.1 Create a yum repo file

First, create a .repo file in /etc/yum.repos.d:

[ phanes@[seekret-removed ] << ~ >>

[- sudo vim /etc/yum.repos.d/silogroup.repo

 

Second, add these five lines:

[silogroup]
 name=SILO GROUP Fedora RPM Repo
 baseurl=http://repo.silogroup.org/Fedora/26
 gpgcheck=0 
 enabled=1

These are pretty self explanatory.  You’ve got a section header, a name, a baseurl, a gpgcheck, and an enabled.

The section header is what shows up in yum as a handle for the repo.

The name value is the descriptive name of the repo.  This will also show up in yum.

The baseurl value is the actual url for your repo.  This can take http, https, and FTP (and I think some others but don’t do that).

The gpgcheck value deserves being described.  This is a way of ensuring the package integrity by the yum client during download/install.  If you gpg-sign the packages that’s good.  If you host your repo via https that’s also good.  These are both things you should do that I will not tell you how to do in this article (maybe next week?).

The enabled value tells yum on the client machine whether or not to actually use the repo.  Handy, amirite?

So check this out, all ye bromancers of the stones, the demo you have been exhaustingly working your way towards, using a client I wrote in python called tenta that I packaged just for this demo:

4.0 Install the package with yum:

Or do it via a configuration management system which is actually still using yum under the hood.  I won’t be for this demonstration.

[ phanes@super-seekret-removed ] << ~ >>

[- sudo dnf install tenta
Last metadata expiration check: 0:03:50 ago on Sat 21 Oct 2017 09:02:35 PM UTC.
Dependencies resolved.
================================================================================================
 Package Arch Version Repository Size
================================================================================================
Installing:
 tenta noarch 0.20171015.91-1.fc26 silogroup 13 k

Transaction Summary
================================================================================================
Install 1 Package

Total download size: 13 k
Installed size: 28 k
Is this ok [y/N]: y
Downloading Packages:
tenta-0.20171015.91-1.fc26.noarch.rpm 46 kB/s | 13 kB 00:00 
------------------------------------------------------------------------------------------------
Total 46 kB/s | 13 kB 00:00 
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
 Preparing : 1/1 
 Installing : tenta-0.20171015.91-1.fc26.noarch 1/1 
 Running scriptlet: tenta-0.20171015.91-1.fc26.noarch 1/1 
 Verifying : tenta-0.20171015.91-1.fc26.noarch 1/1

Installed:
 tenta.noarch 0.20171015.91-1.fc26

Complete!

Super hard, right?  It took longer to write this guide and the guide only took about 10 minutes.

And here’s where the files from the demo package ended up on the system:

[ phanes@sekret-server ] << ~ >>

[- rpm -ql tenta
/etc/tenta
/etc/tenta/nerve.conf
/etc/tenta/tentacle.conf
/usr/bin/nerve
/usr/bin/tentacle
/usr/lib/systemd/system/nerve.service
/usr/lib/systemd/system/tentacle.service
/usr/share/man/man1/tenta.1.gz
/var/log/tenta
/var/log/tenta/error.log
/var/log/tenta/producer_source.log
/var/log/tenta/tentacle.log

I am also able to determine now if a file has been edited since the package was installed, and much more.

Notes

*In Fedora 26, the default Apache configuration will still show a splash page for Apache at this point.  You may want to add a landing page at /var/www/html/index.html just to polish it out.  In my case I just commented the "Options -Indexes" line in /etc/httpd/conf.d/welcome.conf because I’m using a dedicated server for this and literally only want directory listings.

** Not to be confused with Lean Transformation Model, which destroys organizations, their cultures, costs more money than it saves in most implementations and suppresses talent in workforces amidst a new atmosphere of fear and instability while somehow sounding really cool to stakeholders.  Lean is a westernization of the Toyota Production System used in the 1950s in Japan and doesn’t translate well to western business models (and is extremely harmful to software engineering process), but it did create an entire market of consultancy for how to implement it.

*** This is sarcasm.  This is not hard, not complex at all, and it saves you tons of time.

How I use it

In my case, the servers intended to use the packages are controlled by a configuration management (CM) system. The CM’s manifest has an entry to ensure the latest version of the package from the repo at all times.  This will force the machine to update the package automatically.  This is only for my pre-production environment — I version lock it in my production environment manifest to give me a scheduled release cycle.

For preproduction CI, the Jenkins server is set to listen for changes to my git repository’s master branch to automatically generate an RPM build immediately and then pushes that RPM to the RPM repository, and then updates that repository for consumption by the client.

This way I can commit my code, install my code in a clean way that’s auditable and manageable even if I had tens, or hundreds, or thousands of packages going, I can see the changes automatically in my pre-production environment for rapid testing, and I can accommodate reliable release management with almost no effort at all– well, a little effort — you just read all of it.  Oh– and I’ll never have a single mistake in deployment.

If you’re curious what the jenkins job looks like, too bad, but, I can give you this early version to build on that does almost that.  The filename and version are where some magic happens.  You’ll want to choose between timestamps + build number or git revision numbers.

You’ll have to restructure the order of variable assignment if you want the latter (trust me both you and your devs want the latter but this is a freebie).

#!/usr/bin/bash
set -m
###
# PREWORK
###

# RPMBUILD ROOT CONTEXT
TOPDIR=$(rpm --eval='%{_topdir}')
if [ ! -d "$TOPDIR" ]
then
    mkdir -p $HOME/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS,tmp}
    mkdir -p $HOME/rpmbuild/RPMS/{arm7,arm9,i386,i686,mips,miple,noarch,ppc,ppc64,x86_64}
    echo "%_topdir $HOME/rpmbuild" >> ~/.rpmmacros
    TOPDIR=$(rpm --eval='%{_topdir}')
fi
echo "TOPDIR $TOPDIR"

# TMP PATH
TMPPATH=$(rpm --eval='%{_tmppath}')
if [ ! -d "$TMPPATH" ] || [ "$TMPPATH" = "/var/tmp" ]
then
    mkdir -p "$TOPDIR/tmp"
    echo "%_tmppath $TOPDIR/tmp" >> ~/.rpmmacros
    TMPPATH=$(rpm --eval='%{_tmppath}')
fi
echo "TMPPATH $TMPPATH"

GIT_REPO_BASENAME="tenta"
GIT_REPO="git@source.surroindustries.com:ircthulu/tenta.git"

echo "GIT_REPO_BASENAME $GIT_REPO_BASENAME"
echo "GIT_REPO $GIT_REPO"

# The name of the RPM that is being built
APPNAME="tenta"
echo "APPNAME $APPNAME"

# version number for this build
VERSION=0.$(date +%Y%m%d)
echo "VERSION $VERSION"

# The directory containing .spec files
SPEC_DIR="$TOPDIR/SPECS"
echo "SPEC_DIR $SPEC_DIR"

# the target spec file
SPEC_FILE="$SPEC_DIR/$APPNAME.spec"
echo "SPEC_FILE $SPEC_FILE"

# the workspace dir
prearchive_dir=$APPNAME-$VERSION
echo "prearchive_dir $prearchive_dir"

# the source spec file template
PRE_SPEC_FILE="$prearchive_dir/$APPNAME.spec"
echo "PRE_SPEC_FILE $PRE_SPEC_FILE"




###
# WORK
###

rm -Rf ./*

# create a dest dir (ensuring write access to target fs)
mkdir -p "$prearchive_dir"

printf "\n***\n*** OBTAINING AND HOMOGENIZING SOURCE ***\n***\n"

# Get the source
git clone "$GIT_REPO"

# deglove the unnecessary git repo and child dir
mv $GIT_REPO_BASENAME/* $prearchive_dir &&
rm -Rf $GIT_REPO_BASENAME
echo "GIT REPO $GIT_REPO_BASENAME => $prearchive_dir"

# cat the spec file for the log
printf "\n***\n*** SPEC FILE TEMPLATE ***\n***\n"
cat $PRE_SPEC_FILE

###
## SPEC VARS
## These are variables that are expected to be in the SPEC file.
###
SPEC_VERSION="__VERSION__"

###
# Transform the SPEC file
###
mv $PRE_SPEC_FILE $SPEC_FILE
sed -i "s,$SPEC_VERSION,$VERSION," $SPEC_FILE

printf "\n***\n*** SPEC FILE INTERPOLATED ***\n***\n"
# cat the SPEC file for the log
cat $SPEC_FILE

# bundle up the now distributable source tarball to the rpmbuild SOURCES dir
printf "\n***\n*** SOURCE ARCHIVE CONTENTS ***\n***\n"
tar cvzf $TOPDIR/SOURCES/$prearchive_dir.tar.gz ./$prearchive_dir && 
rm -rf $prearchive_dir

echo "Archive location: $TOPDIR/SOURCES/$prearchive_dir.tar.gz"


printf "\n***\n*** RPM GENERATION ***\n***\n"

rpmbuild -ba $SPEC_FILE

printf "\n***\n*** STAGING RPM to Repository ***\n***\n"

scp $TOPDIR/RPMS/noarch/$APPNAME-$VERSION-*.fc26.noarch.rpm repo.silogroup.org:/repos/Fedora/26/Packages/

printf "\n***\n*** Updating RPM Repository Metadata ***\n***\n"

ssh repo.silogroup.org createrepo --update /repos/Fedora/26

So, I’m working on some new approaches to some projects I’m trying to get off the ground.

One is for a client and I’m ethically obligated not to disclose in this context.  We’ll call that project X.

The other is Examplar, which has attracted some new potential developers that found me through the SURRO Linux project at a Linux convention in Ohio this year.  I’ve got a little time as everyone that showed interest is still getting adjusted to using IRC for reliable communications (it actually takes some time if you’re not used to it).

Since we use IRC exclusively for comms, I’m able to be ready for when they’re able to code in advance.  They’re going to want formal spec and design documents, which I respect and agree with.  Simply having them may even attract yet more devs, so, I need to get off my ass with Examplar.

From there I have got shitloads of additional reading to do.  Namely, a revisit of the puppetlab’s training VM from last year that I abandoned took a hiatus from due to frustration over puppetlab’s tendency to rename already existing concepts.

Meanwhile I’m waiting on news for a C developer role that might actually get me the deep exposure I could only find from having to work an FTE role elbow deep in C and only C and it certainly could be my next gig if they’re into me.  Time will tell.

Oh, found out about the plates thing — its just a fine and a fee for renewal.  They dropped the insurance charge because I found the backup paperwork that didn’t get stolen during the repeated break-ins this year (jesus) since they were kept in the house.

They did get my laptop though since it was in the back of the car this last time.  No comment.

And, of course, being on call is this week.

I’ve got too much shit to do at the same time this year and I’ll probably need to cut a project loose so I can focus on the things I like doing.

This content is password protected. To view it please enter your password below:

Password:

We’re in the process of onboarding a couple of new developers that heard about SURRO at a convention recently.  Stay tuned.

As for current priorities, I’ve been distracted with a non-surro related project and work.  Due to not having much developer numbers to pull from development is slow.

It got brought up that we should have a formal spec and design document.  I can’t disagree with that.  To follow.

Super busy lately.

Balancing work with after-work.

Lots to do at work, lots to do at home.

Need to call the DMV.  They’re up to the same bullshit they tried last year.

I need to renew my plates and pay a fine from not being able to renew my plates on their site (bug?).

As it pertains to ircthulu, I’ve got the listener complete.  It dumps everything to a FIFO that a producer is listening on.  The producer is complete.  The producer pumps to an MQ.  The MQ is set up as well.  From there, a multithreaded consumer pulls from the MQ.

The database is set up and ready to receive data.

The consumer will need to be extended to transform the json payloads from the MQ into a format consumable by the database during pushes to it.  Will be pretty easy with JSON, just deserialize to a json object and populate [an SQL insert statement?] with its values while logging failures.

From there I’ll need to create a restful API that will present data from the database for the UI loader designed in a way to easily extend new types of data to present that is consumable by the UI.

This design will allow the clients, the database, and the UI to all be hosted separately from each other in a massively scalable manner.  Scalability was the pain point for the pilot program.

From there I’ll need to build the UI in parallel to the data tier which is hosted elsewhere.  That’s where most of the work for this project will be, mostly because I’m not a UI person.  Yay angular + bootstrap!

Once the UI is ready for curtains to be lifted, it’s just a matter of getting all the folks who will be running listeners coordinated so that if the client bots are targeted it won’t break the data flow.  Everyone’s on board.

This part is especially important because if, for some reason, someone were to start banning the clients, if I were hosting the clients myself I would potentially face some ramifications with my hosting provider, which I will not risk.

In this model I am simply the provider of a data dumping ground and a publishing point for data that is provided to my system by its users.  I will not be logging which clients are owned by who or keeping a catalog of who the providers are.

Upon my trusty workshop keyboard I code:

The designs my trusty pad hath wrought:

 

In good company my planning was invested:

[never insert this image]

For an untold purpose my world necessitates:

[not my image to insert]

On a single node instance of its back end, with 4 clients pushing data into it, and 8 threads pulling data out of it, a performance test was able to send 500,000 messages in 16 seconds and all were processed on the other end on completion of the sending of the test data and it didn’t seem to be anywhere near maxed out.   It is processing 31,250 messages per second.

That’s roughly 30,000 lines of IRC per second that it can handle without having been limit tested which exceeds the needs for this project.

There is still quite a bit of work to do before the fun can even start with the UI which will be a project all of its own.

Next is the client and the ETL on the other end.

 

…You didn’t think I forgot about this project, did you?

or

How not to sell out by AVOIDING HURTING BOTH your career and YOUR client’s organization

I should be working on other things, I’ve got alot on my plate this week, but I wanted to share this for other contractors who may be following.

I interview regularly.   I get lots of calls from recruiters in my area for IT firms.

This time, it was a company called removed.  There were two recruiters involved.  One was aggressive to get me to give approval for the interview and wanted an updated resume.  The other was aggressive at confirmation.

I send it.  They immediately confirm rate before the interview has even taken place (warning sign one).

Some recruiters are just required to ask so I we move beyond that.  Interview time comes along, and I’m about 5 minutes into the interview for a cloud architect opening before the interviewer at the client says “I see here on your resume you have tech stack im obviously going to learn now experience, extensively, can you tell me a little about the projects you worked on with this?”

I say “I did not put that on my resume because it’s not a stack I’m familiar with”.  He says “yes you did, it’s right here”.

I sense something’s not right because he was hyperfocused on a piece of technology previously that I was not terribly well trained on.

“Hold on.  Something’s not right here”.

So I go to the invite and look at the attachments.  There’s my resume.  I open it….

…It’s been filled full of technology I did not put down, that just so conveniently perfectly all line up with the requisition and have absolutely nothing to do with my background.

Maybe I could have faked my way if I’d looked at the fake resume ahead of time, I’m at home enough with this type of tech that I could probably have performed the role with a little lead time to study up on it.

But here’s the problem:  This industry has a faker problem and it’s an expensive problem.  We’ve all seen them.  They somehow make it through the screening process and make every competent person they work with’s life a living hell for months and sometimes cost millions of dollars to a company before being found out, by which time they’re moving on.  They band together at companies when they identify each other and use politics to suppress talent so that they aren’t exposed.  I’ve seen it.  That, and, the client deserves to know they’ve got a resource provider that is screwing them over if they want to stay in business long.

I say “Yep.  I knew it.  I’m really sorry sir, this is not the resume I sent over.”

“It’s not your resume?”

“Well, it is, except, the resume I sent over had different specializations on it.  I am so sorry about this, I am going to have to end the interview and then fire the firm I was working through for this interview.  I will never work with them again and you should be made aware that this has happened.”.

“Ok.  Thank you for telling me.  I’ll go ahead and alert my superiors to this.”

“This is unacceptable.  You’re looking for a removed architect.  I have strong specializations but would not be a fit for this role as it’s not what I specialization in”.

“Yes, I agree.  Well Chris, I hope to see your name on future openings here.”

“You too as well sir.  I’ll be connecting through a different company if I get a chance”.

And we ended it on a good note.

It was hard to do.  That job could have doubled my income.  But it’s not just the right thing for you that needs considered during your career– as an engineer, or an architect, or a developer, you, as a consultant, have a set of obligations to your clients and to your career field.

The core foundation of those obligations revolves around integrity and in this case — cleaning up the industry in some small way.  A company that was being exploited by one of their vendors now knows to look out for dishonesty from that vendor.

Just like you’d pick up an empty soda can and throw it away at the bus station.  I’d expect the same from a candidate that I’m interviewing.

This content is password protected. To view it please enter your password below:

Password:

A more readable journal can be found here.

[ phanes@prime.surroindustries.lan ] << ~ >>

[- cat 2017-09-20_rpm-build.txt 
@@@ rpm build instructions
@@ build an SRPM

@ 2017-09-21

/*  Note:

    An SRPM is a .src.rpm file that contains the SOURCE for an RPM
    in most situations this will also include source code for the 
    application and this is compiled during RPM generation. 

    Pipeline for an SRPM:
	$PROGRAM SOURCE, SPEC FILE -(rpmbuild)> $PROGRAM.src.rpm -(mock)> $PROGRAM.rpm

*/

$ sudo dnf install fedora-packager rpmdevtools
$ rpmdev-setuptree
$ cd ~/rpmbuild/SOURCES
$ git clone git@github.com:cmpunches/Marco.git
$ cd .. && mv Marco/ Marco-1.0 && tar  cvzf Marco-1.0.tgz Marco-1.0/
$ cd ~/rpmbuild/SPECS
$ rpmdev-newspec --macros Marco.spec
/* output
Marco.spec created; type minimal, rpm version >= 4.13.
*/

/*  Note:
    Creating the spec file is actually careful work that is error-prone and requires training and testing.
    You are advised to read the rpm documentation, particularly pertaining to SPEC file SECTIONS and MACROS.

    Note:
    This is much easier with a build system, either as a result of a cmake build or in a larger system like Koji.
*/

$ vim Marco.spec
/*
Name:           Marco
Version:        1.0
Release:        1%{?dist}
Summary:        An HTTP echo service.  Returns the request it received to an HTTP client in a consumable format.
License:        GPLv3+
URL:            https://github.com/cmpunches/Marco
Source0:        %{name}-%{version}.tgz

BuildRequires:  cmake boost-devel
Requires:       boost-devel

%description
SURRO Industries' HTTP echo service.  Listens on port 1999, and replays requests to the requesting browser.

%prep
%autosetup

%build
%cmake .
make

%install
%make_install

%files
%{_bindir}/marco

%changelog
* Thu Sep 21 2017 Phanes <punches.chris@gmail.com>
- Initial version of the package

*/

$ cd ~/rpmbuild/SPECS/
$ rpmbuild -ba Marco.spec

/* compiles and bundles into an SRPM */


@@ Build the SRPM into an RPM

/* initial mock setup */

$ sudo dnf install mock mock-scm
$ sudo usermod -a -G mock Phanes
$ newgrp mock

/* end initial mock setup */

$ rpmbuild -bs Marco.spec

/* Note:
   At this point you've got an RPM and an SRPM for Fedora since I'm on a Fedora system. 
   Probably not desired.
   Mock allows you to compile for other systems.
*/

$ mock -r  <Tab complete for system profiles>
$ mock -r epel-6-x86_64 ~/rpmbuild/SRPMS/Marco-1.0-1.fc25.src.rpm

/* Note:
   Tons and tons and tons of loading.  That's expected.  It's creating a mock build environment for the
   compile so that you're getting a clean package that's aware of the profile of the system it's intended for.
   These packages are not installed on your driver.

   Note:
   Due to a transition from yum to DNF for RH upstream, which has not yet hit EL6 and EL7, you will need
   a newer version of mock that fixes a bug caused by that change.  It is in the updates-testing repo.

	sudo vim /etc/yum.repos.d/fedora-testing-updates.repo

   Enable it, then `sudo dnf -y update && sudo dnf update mock` 
   Note:  I also had to install the undocumented package yum-utils to bring the /usr/bin/yum-builddep tool back.

   At this point I had to reduce the boost requirement to 1.4 to compile on EL6 which won't compile without a
   full rewrite of my solution since it heavily uses the boost libraries.  EL6 is old.  Trying EL7.
*/

$ mock -r epel-7-x86_64 ~/rpmbuild/SRPMS/Marco-1.0-1.fc25.src.rpm

/* EL7 is also too old for the version of boost used in the application code.  Will have to revisit this for EL distros. */

@ 2017-09-21_2

/* Let's go ahead and bundle the boost dependency too. */

$ cd ~/rpmbuild/SRPMS
$ wget http://download.opensuse.org/repositories/home:/aevseev/CentOS6/src/boost-1.63.0-1.9.el6.src.rpm

/* generates a spec file */
$ rpm -i boost*rpm

/* compile to FC package */
$ rpmbuild -ba  boost.spec

/* yay, more dependencies: 
[ phanes@prime.surroindustries.lan ] << ~/rpmbuild/SPECS >>

[- rpmbuild -ba boost.spec 
error: Failed build dependencies:
	libproxy-devel is needed by boost-1.63.0-1.9.fc25.x86_64
	openmpi-devel is needed by boost-1.63.0-1.9.fc25.x86_64

$ cd ~/rpmbuild/SRPMS && wget http://fedora.inode.at/updates/26/SRPMS/l/libproxy-0.4.15-2.fc26.src.rpm

$ wget http://mirror.math.princeton.edu/pub/fedora-archive/fedora/linux/updates/testing/24/SRPMS/o/openmpi-1.10.7-1.fc24.src.rpm

$ rpm -i libproxy-0.4.15-2.fc26.src.rpm 
$ rpm -i openmpi-1.10.7-1.fc24.src.rpm 

[ phanes@prime.surroindustries.lan ] << ~/rpmbuild/SPECS >>[ phanes@prime.surroindustries.lan ] << ~/rpmbuild/SPECS >>
[- rpmbuild -ba libproxy.spec error: Failed build dependencies: libmodman-devel >= 2.0.1 is needed by libproxy-0.4.15-2.fc25.x86_64 pkgconfig(javascriptcoregtk-4.0) is needed by libproxy-0.4.15-2.fc25.x86_64 pkgconfig(libnm) is needed by libproxy-0.4.15-2.fc25.x86_64 pkgconfig(mozjs-38) is needed by libproxy-0.4.15-2.fc25.x86_64 python3-devel is needed by libproxy-0.4.15-2.fc25.x86_64

 

There’s a better way to do this I was just hoping to have a clean build tree for boost.  I’m just too fucking tired, I’ll have to carry this over to a third day until I can get some time to myself.

Moar diagrams

This time covering service state control patterns:

So from here I’ll need to create an outline, and then condense everything into brief, concise writeups that fit into their appropriate sections.

As for service controller antipatterns, I just don’t have alot of those besides what was already covered.

It’s going to be difficult to justify the reasons why in these, so, I’ve got a good couple of days more work ahead of me.

Update:  These need a change — variant A and B for established control patterns needs updated to reflect that it requires a root setup for enabling user services but does not require root or sudoer to actually consume.

Speaking of Work

In other news, I’ve hurt myself at the gym.  I have some DOMS after hitting curls too hard too early and can’t fully extend my arms.  I’m basically a whiny T-Rex until it’s healed up.  I had to actually call in and sleep for a good day and a half.

Welcome back.

I said I’d post this “tomorrow” in my last post, and it definitely has been several “tomorrows” since then (mostly because I’m lazy and busy, which is a terrible combination), so let’s recap:

This series of articles revolves around:

• The reasons why packaging any software used in your project’s environment is important.
• Service management considerations for your software on the Linux operating system.

The approach we’re taking emphasizes where this would take place in the SDLC, specifically the ALM (Application Lifecycle Management) which I encourage you to read up on if you’re not already familiar with these entrant software engineering concepts.

To do this, we’re removing all the coding, all the solutions architecture and design out of the software engineering process so that we can look at just these two aspects — and the example that we’re using is an HTTP echo service that I’ve hacked out of a very simple HTTP server in C++ with the boost library to create something called a microservice if you’re familiar with SOA (Service Oriented Architecture).

These are all suggested google search terms.

But first, what genre is this even?

Anyway, to bidness, some patterns and antipatterns as it pertains to package management (I’ll be breaking this into cleanly organized content when it’s all laid out btw), I’ve started some crude graphs that demonstrate what I’m actually saying to be used in a more meaty article later.

Package Management Is a Fundamental Best PRactice

First, we’ve concluded that packaging your software is an intrinsic result of abstraction of software to units on your system using a simplified representation of parts of the OSI model:

Service State Management Patterns and AntiPatterns

Patterns

Variant A and B

Antipatterns

Examples A, B and C

Given the amount of research it takes to justify the reasons why sysadmins are trained to do this in this way, I’ll need to spend more time on the meat of the article and will end up condensing this whole series into a whitepaper.  But for now, I need the rest of my Saturday.

Howdy.  My name is Chris Punches (vanity link), and I’m here to talk to you a little about the planning around the Deployment Phase and Maintenance Phase in the Software Development Lifecycle.

# BEst Practices – I want to Believe

To preface, over the years I have seen good habits that save lots of work for everybody, and I’ve also been brought in a time or two after damage has already been done by bad habits — often either for the sole purpose of cleaning it up or making recommendations on what to do. I’ve also dispelled some bad habits that “everyone was doing” and those are plentiful in the industry as well.  Basically, I’m a software, systems and infrastructure janitor, and I try to share what I learn where I can or document my own learning for others.

Let’s talk about the SDLC, but let’s remove the SD from it

So, for this example I’m going to demonstrate by removing the parts we already know about from the SDLC: The actual design and development of the application. While it would seem at first glance to be important to talk about software development in a talk about the Software Development Lifecycle, that part of the lifecycle actually so dominates most environments that conversation about the rest of the lifecycle can get drowned out:

In this example, I want an HTTP ECHO SERVICE that will simply return 200 to a browser along with the headers supplied in the original request. We’re going to skip design and coding, and omit getting it in version control but assume it will be and that all best practices in software engineering were followed up to this point.

Note: This will nicely compliment my proxy discovery tool that I’ve recently created. It allows you to verify that a proxy on the list is not tagging users with an injected header (something that will certainly compromise the identity of users of that proxy if not checked for before using it).

So in this case I got kind of lucky right after I started and found that the boost library documentation in C++ has an example of a very small HTTP server, that I just kind of hacked up to regurgitate whatever request was supplied to it instead of sending an HTDOC root path and doing all that fun stuff.

Annnnnd done:

Then an hour of goofing around with gitignore, building the readme, creating the public repo, and we’re at about the point that too many folks stop developing because they think software development only involves the development of software. I actually enjoyed building this so much that I’m going to host it as a permanent service at, say, proxycheck.silogroup.org and I’ll put the source here.

# How to get from “it compiles” to “I GET PAID TO DO THIS”

So here’s the meat of what I’m writing about. The software itself is only one piece of a much larger set of interacting components in your SDLC, and some of those components are the people involved, as well as the time they spend on it, and all the things people have to do to coordinate with that in an operations environment. If I were, say, hosting it in an organization (which I am, SILO and SURRO), then I would have considerably more things to consider than if I were just testing it on a VM or showing a friend.

That’s probably all some folks are used to besides a “sudo make install”, which you can get by with on a disposable machine, but not for actually hosting a service reliably, and will get you shot if you do in enterprise.  That’s almost as bad as compiling on one machine and just copying the assembly over (READ:  PREPROCESSOR, COMPILER, ASSEMBLER, LINKER, LOADER).

If you are out there reading this and writing software, then definitely don’t stop here because this is where it gets cool.

# So how do we get there knowing at least *someone* will want to:

• make updates to this service’s source code that I will want reflected in production
• update the installed version to newly compiled versions after that
• troubleshoot it in familiar, reliable ways when it breaks (assume bugs exist)
• report on the assembly metadata as supplemental process to that
• control the virginity of files associated with that piece of software (hehe i said ‘virginity’)
• control the service’s state, particularly during installation, upgrades, and removal of the software (too many people forget that last one and end up breaking their systems when they’re first learning stuff — think:  library conflicts).

These are great initial considerations if you actually care about what you’re doing or if what you’re doing involves other people.

# Enter the OS (+Userland)

First you have to understand that you’re controlling a piece of software’s lifecycle, and that software requires a platform: The Operating System.  It is the platform upon which software runs.

This is where alot of people get tripped up because it’s easy to forget what the purpose of things are in complex systems after we’ve seen them for a while and don’t use them.

The operating system is the interface between the user and the hardware. Everybody knows that.

Here’s what alot of people don’t know though: It consists of a kernel or userland components.

Userland is everything on the OS besides the kernel.

Userland is divided up into three categories: (on Linux Systems)

• The C standard library,
• Low-level system components,
• and user applications.

It is similar on Microsoft and Apple systems.  System daemons running, say, systemd, logind, PulseAudio, and other low level stuff fall into low-level system components. User applications will be things like your browser. Apache is a system service. Tmux is a user application. We’ve built a system service because:

1. It’s always running.
2. It’s meant to be a background process.
3. It’s unix-like.

Learn more about daemons, or services here.  The criteria for a system daemon are well defined and understood and I have no need to justify them here.

You don’t handle these processes that meet the needs discussed in the previous discussion in the same layer as your application if your operating system will be running other software that other people write and support.  But why?  For the same reasons you don’t leave a mess in the kitchen when you have roommates first of all, and in the case of organizations you’re saving time and money due to uniformity of process (oh shit now I have to justify why uniformity of process saves money — ya know what, fuck it — it does, just believe it for now) and reduced human error (it is a form of automation to behave in this manner but it is so common now that people don’t even realize it). But, mostly, it’s because of the importance of abstraction:

# Wtf is abstraction?

If you’re a developer, it probably means close to what you think it does, but is not an inherently development-centric term — it’s also an architectural term (and has a similar meaning in several unrelated contexts):

“In software engineering and computer science, abstraction is a technique for arranging complexity of computer systems. It works by establishing a level of complexity on which a person interacts with the system, suppressing the more complex details below the current level.”

So, in this case, there are some revised points of consideration to point out that are solved by abstraction of the software artifacts:

• We want this service to be portable to a wide number of machines ( this allows a rational distribution model for this piece of software and also prevents conflicts with other software on the providing platform )
• We want to be able to deploy, update, and remove cleanly ( this reduces outages of my new service before, during, and after updates and installation )
• We want the above to be done in a uniform manner to other software operations on those same machines ( this reduces human process surrounding unit management on the target platform, allowing one process to exist for all software in a way that’s able to be controlled and improved; this also reduces human error by natural consequence of that, reduces the amount of support training needed on this service, etc.  )
• We want to minimize the amount a human hand has to touch the unit or the platform serving the unit.  Be easily updateable without potentially breaking things or creating inconsistency in the environment we’re installing to –not touching too many “things” during operations involving it ( this should include maintenance of whatever mechanism or automation is being used).
• We want to be able to audit both the assembly and metadata of all associated files during troubleshooting and analysis, even planning, and this metadata should include at minimum version number, dependency trees, install date, and more if we can get it ( while more complex than this, it basically allows us to know

So we need to Abstract it into a unit.  How does this work?

Artifact abstraction in computer science is widely regarded as packaging the software, and sometimes in several layers of packaging.  The definition of packaging is literally abstraction of a software unit.

If you’re in Javaland you’ve probably used Artifactory for this.  If you’re not, you’ve probably used MSI if you’re on windows, or RPM if you’re on CentOS or another redhat derivative, or DEB/APT if you’re in Debian or Ubuntu.  If you’re on Arch Linux you’re probably using pacman.  Gentoo uses emerge (it technically is package management even if it doesn’t feel like it in gentoo).  There are distro-independent package managers of varying qualities.  I’ll not mention them because they generally tend to bring instability to systems due to bugs.  If you’re on OSX apple has their own package management scheme, it’s actually so seamless that it is barely noticeable — however they also have alternative package managers like brew.  If you’re on a freeBSD system, you’ll probably be using pkg_add which is built-in and portsnap.  If you’re on SUSE it’s zypper unless that’s changed (been a few years).

In my case I want to target for Fedora since that’s the distro I use on all my servers currently.

In the case of a robust package manager like rpm or dnf, the process of installation generates associative metadata in a repo for every file in the package meeting every need presented but one.

I’ve actually rarely had to justify package management though, it’s really such an introductory concept and is so fundamental to software management that almost invariably doesn’t need justified.

This is not an operating system concept though — it’s actually a software engineering concept, a fundamental one, that the OS makes heavy use of.

Reiteration:  The Linux Operating System does not provide a package manager.  The project that develops the package manager provides it to the distribution maintainers who decide to include it or change to another package manager.  This is easily confused because often the team for a package manager that is widely used by an OS works in close collaboration with that distribution provider.

On to Daemons and Services

So, I didn’t solve every need that I have for the actually very important “people” part of my SDLC:  I need a uniform way to control service state for all system services, not just for this one piece of software, but for all software I put on that system.  I don’t care if it’s in the base packages the distro provides, or if it’s something I put on after the fact — I’m building a complex set of IT Services out that I want to have clean management of and this daemon is no exception.  Since in the case of SILO and SURRO the same people who manage the state of this service will also be managing any network services or anything else on the hosts, it’d be much better to have it all using one tool.  If I don’t have a uniform state management facet of this service’s lifecycle, the silo staff and surro staff will be running around pulling their hair out, being well trained on linux systems, trying to figure out how to start which service with which methods.  After about 10 or 20 disparate services like that, it can become almost unmanageable even if you take copious notes unless I allocate staff to only watch those services and nothing else (and I can’t allocate people to just OS maintenance because my application layer is deeply dependent on it and I need them to support that user application layer as my core organizational goal).

But first a word on service state.

# Service State is a Lifecycle Too

If your daemon is running, that’s a state.  If it’s not, that’s a state.  Here are all the states I care about:

1. A Running Process,
2. No Running Process,
3. A boot-enabled process,
4. A boot-disabled process.

Services are crazy simple when you look at service states.  Just like processes.

Now sure, the application itself may have many states within it, an uncountable number of states — but those are up a layer in this model and I don’t care about them yet.  At this point I care about the state of the service itself.

Oh, I should also point out– a consistent method for reporting on the state of the service is necessary as well, as during troubleshooting I don’t want to wonder at all which of these four states my process is in (note: not all of these states are mutually exclusive) if other services will depend on it (and they will in my case).

So how do you bring continuity to service state management?

I’m glad you asked, very large header text.

It starts by understanding how your software is launched.

In all cases, including Windows, and Linux, and OSX, and BSD, your server starts with a single process bootstrapped from BIOS/UEFI.  That process is called init.  The kernel itself executes init as the bootstrap mechanism with a hardcoded path to the init program.  It gets complicated and divergent from here but basically init calls other processes.  In the case of Fedora, it launches SystemD, which then starts the system’s services that allow it to become multiuser, and sets up various features that most people don’t dive very far into.

From there you log in, and do things.  Kinda.  I skipped over alot of stuff, but that’s the gist you need for this.

Once it’s able to handle users, I can log in and use it’s reporting and control tool, systemctl to:

1. Start a service that is not running or stop a service that is running.
2. Report on the service state of any service.

And there you have it.

So, if I want a uniform way to manage system services I would plug right into what’s bringing up the rest of the daemons running on that server.  Even if I don’t — by proxy of another process I still am anyway — it is not something you can opt out of.

• We want to be able to control the service’s state cleanly.

And for the final point:

Oops, changes of service state with systemctl require root and I don’t want to give root to anyone.  SO DOES INSTALLING PACKAGES.

This is a perfectly valid concern.  As it should.

So, we’ve made it this far only for SILO security staff to blow the whistle on my design because SILO’s servers require more than one person to have access.  What will I do?

 

Well, here’s where it can get a little complicated but it’s definitely a solved problem.  Since I’m not incompetent I set up all SILO staff to have users in a shared group called “staff”.  Granular access control is the point of usergroup administration.

Then, I divided the staff into users who can install software and other users who can manage service state.  One subgroup is “installers” and one subgroup is “sysadmins”.

Then, I added two lines to a frequently misunderstood file called the sudoers file.

%installers ALL = NOPASSWD : /usr/bin/rpm
 %sysadmins ALL = NOPASSWD : /usr/bin/systemctl

And from that point forward, my staff who installs things has the access they need without root.  And my staff who controls service state have the access they need, also without root.

Or,

I could have them use rpm to install to a shared system path that they do have access to, like /opt/SILO/RPM_ROOT and force approvals for service scripts, though, that can get very complicated quickly.

In my case, I opted for the sudoers entry because it’s very easy to manage and sudoers leaves an audit trail with logs that I have pumping into my monitoring system and SURRO pager.

Even better news:

Packages are as easy to build as whatever software I am packaging is.  This is also usually automated, and usually in a CI-like system.  Check out koji or git-build-rpm or Jenkins if you like getting your hands dirty with the build process.  Having a package regenerated when I do a git commit is excellent as it can immediately reject if it doesn’t compile.

Systemctl is stupidly simple to use.

I hope that this report saves readers a great deal of money and time — from reallocated time spent automating things, to outage prevention, to general system stability and reduced error rates.

Implementation

In my case I’m doing a one-off so I won’t be using a CI model although I will be eventually for SURRO.  With this package I’m going to just create the SystemD script, which is all of about 4 lines, build an RPM, install that, and then open the firewall for my new webservice.  I unfortunately won’t have time tonight to do that though, so it’ll have to be tomorrow.  This should cover all the important parts of the lifecycle you need to know — I don’t need to document very much about my service now since it’s all using tools that I’d not let someone be staff at surro without knowing how to use what I mean by that is that while the explanations and reasons behind these choices may have been rather complex, the practices that were mentioned as a result of that thinking are entrant level exposure.

So, I’ve been lazy.

Apparently during the move to HTTPS, I failed to set up some things.  The personal blog is still loading http for media assets and RSS, but http is no longer available, and one of our certs expired because i didnt properly configure the alias for the automation behind cert regeneration and then also didn’t set up a monitor for that.

I’ve work to do.

NEWS Outlet is now Mobile Friendly

Did give the news outlet a facelift to make it mobile-friendly, though, so, there’s that.  It’s still http though.  Gotta get on that.

TODO

• move news outlet to phanes.silo server
• set up automated cert renewal for news.silo
• regen certs for phanes.silo and news.silo
• fix https in application layer for phanes.silo
• update news.silo to point to phanes.silo
• set up LDAP auth and UID/GID mapping for SILO

LDAP for SURRO and SILO

I’m also in the process still of finishing up configuration of a new LDAP service for everything.

So, it’s SSSd, NSSwitch, and PAM, yay.  3 huge parts of the linux system I never really made myself learn.

I think starting out with Kerberos 5, SLAPd, LDAP working my way down the docs is the smart way to go there.

The fun part is Ipsilon after that but I want to get system auth up and functional first.

Reading for Network Login SErvices

So, when it comes to KRB5, SSSD, NSS, and PAM, this is an area of linux systems that everyone skips over (myself included) because there are just so many moving parts and it’s a pain in the ass to troubleshoot.  But it needs learned if we’re going to implement it in a new distro.  We’re looking more and more like an enterprise infrastructure every day and I need to get us to “that magic quiet zone” after all the shifting around.  I find this part of the system to be far more complicated than SystemD and even compiling a custom kernel.

• The TLDP Guide for Kerberos.
• RFC 1510 for LDAP definition.
• An Ubuntu Guide (I’m using Fedora but it’s helpful) for Kerberos + LDAP.
• The official MIT-origin Kerberos Documentation
• SSSD and NSS docs from Red Hat
• SUSE’s expectedly sexy and clear presentation on SSSD.
• An introductory text on how PAM works from Tux Radar.
• The official docs on PAM.

Examplar

Still plugging away.  Pipes, pipes, dupes, forks and more pipes, oh my!

New Rack SErver and 32bit compat in python

Someone donated a new rack server for lair.silo.  I’m not sure what I’ll be doing with it besides hosting auth services since it is 32bit and can’t be the archipel demo (and can’t really be preprod for any of the python services since 32bit py environments don’t seem to have library continuity with 64 bit in just the right cases).

Oh well.  Tons of shit to do.  Time to go work out first and then get to it.  With it being labor day weekend it’s the perfect chance to get caught up.

Wow.  What a great vacation.  It’s been, like, forever.

In terms of stuff that would be relevant for this blog, not much.

Well, I did build an interesting little tool for a browser anonymization experiment:

HOWDI – Http Operations With Dynamic Input

https://github.com/cmpunches/HOWDI

Basically is a stripped down rewrite of SlitScan only a little more usable (and maintainable — there were serious design quality issues with SlitScan that I found and wanted to just reimplement almost all of it).  Needs a posix interface though.

How HOWDI works

It fetches proxies from open proxy lists.  It then checks from a random selection of IP address echo services using each proxy in the list with managed concurrency.  It then creates a listener on localhost that routes to each proxy until a side disconnects.  It is tested and working but there’s a pretty gnarly file descriptor leak somewhere in the socket code for the L4 proxy portion.  I’m not entirely sure where yet.  Socket programming is a big gap for me, so I look forward to researching it.

Joined a Gym

I decided on my awesome vacation that I’m spending too much time in front of a screen.  So, I joined one of those super fancy ‘everything in one’ gyms.  Feels like a cross between a Y and a country club.

That first workout was rough.  It’s always the cardio that gets me.  The pool and steam room were a nice change and the building is absolutely gorgeous inside, where the staff are fantastic.  It’s like a hotel.

Migrated this server

I migrated the blog and some various IRC projects over to a new server.  You’ll notice in the URL when you go to blog.surroindustries.com it now is a CNAME entry for phanes.silogroup.org.

Examplar

Still pecking away on how I want part of process execution to work.  It’s weird with this project so much more time is spent on design than coding.

Ivory Tower

I’ve gone ahead and released Ivory Tower into alpha.  I don’t think I ever announced it.  Ivory Tower is a GTD system for Linux desktop written in GTK and Python.  Feel free to leave pull requests or requests for changes, or file bugs.  A bit rushed but everything is.

IRCThulu

Designs still in draft.  I’m not sure how I want deduplication to work yet.

And a bunch of other stuff I probably forgot about.

 

Not a whole lot of updates, I guess.  I’ve just been working alot.

The last week or so I’ve been researching cryptocurrencies since it’s been a bit of a gap for me.  I have an idea I’d like to try out using microtransaction calculations on cryptocurrency exchanges based on predictive trend convergence when I can decipher GDAX’s or some similar Forex-type hub’s API well enough to create trade logistics algorithms.

Then there’s message queues.  I’ve long talked about implementing my own very simple message queue.  But, I think it’s time to get a little more experience with existing MQ models so I can see if there’s any feature bloat I may actually care about whenever I do find time to do that.  To be honest I can’t take on any more pet projects or I won’t get anything done.

Examplar subprocess execution is fleshed out but not yet implemented.

There was a suggestion to use Yocto for Foster but I’ve decided not to go with it as it would push all the real work from M2 to M3 and the type of work it is needs to be burned in by M3.  It certainly is on the radar for projects to learn from though.

And then there’s also crushing my enemies, seeing them driven before me, and hearing the lamentation of their women.  Work in progress.

 

 

I am pleased to announce the beginning of a long-mentioned, but previously shelved project: IRCThulu. This is a product that was demonstrated during its proof of concept phase in August of 2016.

This project is able to serve as a means to providing a very informative resource to the public as a source of unending Q&A in discussion of software development, linux systems administration, and much more — between developers and engineers— as they develop and do complex tasks that your team may be facing.

IRCthulu is generally oriented towards IRC operators due to its unique identity matching features that will allow you to track users in ways not generally available today, and is not intended to be a public site hosting tool, though it certainly could be if you’re wanting to make your IRC channel public.

I am now allocating all spare funding, time and resources to the development of IRCThulu from here until the foreseeable future. While SURRO Linux will be getting top priority still, I’ll need stuff to pick away on as a distraction.

Bear in mind that IRCThulu is only developing the components necessary to accomplish the capabilities it provides and that other parties have committed to hosting it when it becomes functional.

That in mind, as part of phase 2, as of today, July 31st, 2017, all content on all public python IRC channels sponsored by the PSF are now to be considered in the public domain. It is advised of any channels with any privacy concerns caused by this to alert their users regularly and immediately that their IRC channel, including statements made by users, and their connection metadata, are now published data in the public domain and are google indexed. The reason for the importance of this is that with the introduction of this project they are moving from a closed, private mode to a public one.

Any member of the PSF is welcome to reach out should they have any questions, concerns or comments.

Development begins immediately. This will be a closed source project until the goals are met, at which time it will be released with a GPL license along with the source.

365 Days of Total Transformation

Man, what a crazy and productive year this last year’s been.

The career changes I made a year ago and the extracurricular strategies I formed have started to burn in enough that it’s completely transformed my home life, my work life, my general outlook, and desire to be surrounded by puppies, for the better in a much needed way.

I’ve broken a deep, decade-long depression in the process and I can not be thankful enough.  Groceries from Whole Foods.  Vitamins, exercise, sunlight, diversity in activity, respect among my peers, and consistent growth.  It was that simple.  Oh, and new furniture.

New furniture helped.  It was weird how big of a difference that made.  There is something about that I don’t fully understand.

I’m ALSO Taking Downtime as an R&R supplement

I have to confess, I’m in a “downtime” mode right now and I’m so not sorry.

I did manage to get to an alpha state for Ivory Tower.  It is not quite ready for release but serves as reasonably accurate prototype for a C or C++ rewrite where the library bindings are little more sane.

I just got back from a python convention, which I missed most of unfortunately, this year, due to being on an outage call for my primary client.

Speaking of Python and Bullshit

And now I will indiscriminately throw lots and lots of people under the bus.

Speaking of Python, being reasonably accurate, being sane, etc, etc, etc, my attempts at negotiating for a reasonable compromise with the PSF around IRCthulu seem to have failed after a year of extending innumerable courtesies to reach a sustainable agreement.

I don’t know what’s going on with the PSF these days, but I’ve never seen the kind of behaviour from a respectable organization that I’ve seen surrounding the currently shelved IRCthulu project as I try to establish a reasonable relationship with that team.  I am not impressed with their ethical perspectives and I am easily impressed in that area.

It seems I am not alone in this regard.  It is concerning.  I think that I’ve reached a point where we may be at an impasse in negotiations unless they can prove to me that they are sane fucking people able to have sane fucking thought processes.

I’m at a point in my life where I’ve got just zero patience for manipulative assholes in my mindspace anymore. Between Red Hat’s oppressively idiotic influence in the open standards area and PSF’s absolutely pretentious views from their IRC teams, gnome’s arrogance in the Ux and DE crossover compatibility areas, AMD’s archaic and mostly retarded stance on chipset instruction disclosure in combination with their inability to package their own fucking software for their target platforms,  Apple’s poisonously destructive propaganda surrounding their efficacy as a provider of sane development environments, Oracle’s refusal to die or shelf Java, IBM’s and HP’s incredible ability to repackage open source systems in an unmaintainable and unnavigable way to secure support contracts behind the veil of lack of client training, and the continued fucking existence at all of lotus notes I’m just remarkably disappointed in the times I live in sometimes.

Then I see articles like this.

About the machine learning algorithm that used improper English to shorten sentences due to a lack of proper constraints being set, where it is labelled “AI”, given spooky undertones to misinform viewers about the nature of what was learned (which wasn’t really learned as they already know the language constraints need calibrated to avoid this).  Some of the downstream publications about that project allude to it being “shut down in fear of its capability”.

And it’s utter horseshit.  Nobody who understands how those tools are built thought that.  Nobody involved in the project thought that.  It’s just a lie.  It’s not really AI.  It’s a borked function call to a neural net for predictive decisions.

This concludes my rant about the moral and cognitive erosion in my field of passion.

End of Bus's Danger Zone.

But most of All, I hate Arrogance, especially Mine

And now I will throw myself under the bus.

Anyway, out of all the stupid things I see, arrogance is the worst one, and probably the one I dislike to see the most because it is so easy to fall into.  Other peoples’ arrogance rubs off on the people around them and it can be so infectious.  I’m guilty of this.  I think this is something that anyone working in the software engineering or IT field can fall into because it is just so prominent.  I actively try to catch myself before I let it out, but sometimes it gets out and then I feel like an ass for a while when I realize I’ve been arrogant.  I did that last night I think after 2 too many beers knowing I can’t drink anymore for health reasons.

I think the reason this field is so prone to that is based largely on the fact that it is such a technical field.  The levels of complexity that developers and system administrators deal with on a regular basis gets normalized in their worldview, and there is a wide range of skillset and capability among performers.

There is also a shadow of a problem of pretenders or fakers that shows up occasionally and I think this is the complicating factor; I myself often during screening spot fake resumes or fabricated knowledge during interviews, and have even found some in the past after they were brought on, frequently enough to be a little on the paranoid side, which is both a positive thing and a negative thing– I also spend alot of time on IRC which is an absolute cesspool of human hubris.  There are two points of consideration over why this is a negative thing to me currently:

1. Everybody’s got gaps they’re actively trying to fill.
2. Gaps don’t mean someone isn’t competent in their field

You literally can not learn it all in this space if you spend two lifetimes studying it.  Where one person’s strengths overlap another person’s gaps I think is where the most likely place to see arrogance surface is.

Dev vs. Sys is my favorite example of this because it’s the easiest to describe for me.

Developers can be amazing developers but maybe have some gaps about operating system or deployment architecture depending on what type of developer they are, and System Administrators may be incredible sysadmins who run a clean shop but maybe aren’t so great at development.  DevOps is kind of in the middle by trend, a little of both but not quite as deep in one or the other.  Developers and Sysadmins often are going to have entirely different approaches to solutions engineering or architectural approach to solving a problem, and that should be expected.  This was really highlighted when the devops concept started being ingested by large corporations.  I think often that some devs will not see that or some sysadmins will not see that and it can create misperceptions that create assumptions and ultimately an air of arrogance towards the other.  I got lucky and ended up at mostly great organizations with cultures that none of this applies to, but I’ve seen it before.  When people disagree, everybody needs to be able to be wrong and that’s hard for very smart, technical people to be when their livelihood depends on their expertise, credibility, and ultimately, their delivery of product.  A personal brand.

That, and early IT culture was a little rough around the edges.

Most large orgs in general are aware of this to some degree and have done well in striving to move to something better, and I actually deeply enjoy the direction that’s headed by trend.

Bringing the Dev/Sys example back to arrogance, I was actually a pretty arrogant guy a few years back and actively focus on managing it because I tend to be a little more driven than most and get frustrated in the slow lane.  Every once in a while it creeps up and I learn something every time.  I’ve found through careful self observation over the years that usually when I have to fight that the hardest is when I’m around arrogant people.  Most people who come across arrogantly seem to me to be the same way.  I’m still deciding if it is our shared competitive natures or subconscious aggression levels, or if it’s just some defensive mechanism that people unconsciously learn from each other.  If it’s the latter, that’s almost kind of sci-fi, like, a thing that grows and evolves as it gets passed from person to person and matures in interactions — almost like it has a life of its own.

Ah well.  I’m not caught up on Game Of Thrones yet.  Examplar?  Nah.

Maybe next week.

Home Machine Refresh

or

Full Disk Encryption and Me

Did some neat stuff over the last few weeks and didn’t want people to think I was dead — I’ve just been hella busy.

First, I’ve finished my transition back to Fedora from Ubuntu.  I’d always been a little hesitant about Ubuntu because of the stigma associated with it but I found it to be incredibly well put together and a top notch development environment — and found that moving to Fedora was more a matter of utility than preference due to how dm-crypt / LUKS configuration worked in the installer.  I probably could have pre-worked the partitioning and re-installed ubuntu during the refresh, but opted not to.

Not to say that F25’s encrypted LVM setup was glitch-free.  It was painful and it took hours to get a deviation from the default options to install due to a bug in how pre-existing EFI partitions are [not] detected by it — which was easy compared to Ubuntu’s strange, strange tendency to mount (and fail to unmount) the drives you’re trying to rewrite the partition tables to.  Absolute madness.

Overall, I’m very pleased with the end result.  My machine has a dead man switch for the bootloader and an additional key required to mount anything.  Due to the configuration in use, if I lost the key or deadman switch even a large intelligence agency would likely be unable to recover the data in a usable state.

I’m back in myi3 environment because it’s comfortable to me, but, I am eyeballing awesomeWM.  Not sure yet if I want to make the switch as I’ve just started to explore compositing managers in i3 and have made some neat eye candy for the new setup.

MFG-provided GPU drivers

I didn’t test Fedora’s proprietary graphics driver support because I’m so burned out on that game.

A Super Cool Tool for Reading RFC’s

Found a neat tool that should be, but, alas, is not in the Fedora repos, and I have no idea why after using it.  By bfontaine, it’s aptly titled rfc.

https://github.com/bfontaine/rfc

This is going to be immensely helpful in quoting RFC’s nobody knows I read for fun and profit even though I repeatedly cite them.

And here’s a screenshot of the existing rig:

Obviously quite alot more polish to go as the logo from lightdm is being superimposed onto my background, and I’m not sure why, and it’s annoyingly right on top of my carefully positioned skull and crossbones, which is an absolute necessity for my wallpaper..Not to mention font rendering, while really quite good, is still a little raw for what I like.

I’m using the i3-gaps branch of i3 with i3blocks as a drop-in replacement for i3bar, with (currently) xcompmgr as a compositing manager until I can find one that isn’t exclusively for wayland and also isn’t unstable.

SURRO PROJECTS

As for SURRO projects, I’ve been picking away at Ivory Tower a bit considering a full re-write as a break from Examplar so I don’t burn out.

-CP

This content is password protected. To view it please enter your password below:

Password:

I’m feeling pretty lazy this weekend.

Last weekend was a 4 day weekend and I got a chance to finally refresh my main driver.

I ended up going back to Fedora.  Fedora 25 is exceptionally well put together compared to F24, and this time around I’m running on a GTX 1080 so no fglrx issues (what a trainwreck that was).

I’m taking a break from Examplar to avoid burnout.

This weekend I’ll be picking away at Ivory Tower, a GTD task management system I’ve been conceptually rubbing for too long.

 

 

I forgot to mention, a minor announcement is due.

Freenode has officially recognized the SURRO Linux project as a primary topic group.  It took alot of work to get it to go through.  Thanks to all those who did this work as it really was quite time consuming.

Our primary channel is irc.freenode.org/#surro-linux

-CP

Adulting.  What a difficult thing.

I’d say they don’t tell you when you’re a kid that being a grown up is not all it’s cracked up to be, but they do.  They tell you repeatedly.  I remember many times my father telling me to slow down and enjoy my younger years.

I’m kind of glad I didn’t.  Life wants you poor and humiliated.  If I had listened I’d be in an absolutely desperate situation.  I didn’t have many cards to play besides opportunities that weren’t really opportunities.

That’s the essence of adulting I think sometimes.  There’s a game we all play where we pretend we played the right cards, or just didn’t play the right cards– that the cards we have to play affect the outcome of the game.

They often don’t.  Humans are communal, emotional creatures first — and emulate objective, logical beings as a result of emotional needs to feel that way arising out of communal, emotional reinforcements of culture during development and adulthood — while simultaneously being completely unaware of it.  If you want to beat that part of the game I’m pretty sure you’ve got to avoid playing it at all costs while appearing to play it.

In fact, most decisions that humans make are made before they decide on a reason for their decision:

Neat tangent: https://bigthink.com/experts-corner/decisions-are-emotional-not-logical-the-neuroscience-behind-decision-making

As for adulting, tomorrow is it for sure.  Actually the next month is it.

Part of being an adult is pretending you don’t know sometimes.  But mostly, it’s putting on a suit and tie, shining your shoes, and doing the best with what you’re allowed to use.

As for the personal accountability aspect, there’s always influence.  Touchy topic because it can be confused with social engineering, which is slimy to me.

I see the sphere of influence of a person as having three surfaces:

• Things you have direct control over.
• Things you don’t have direct control over but can influence.
• Things you can not influence.

Things in this context include your tools.  These are the things you use to get tasks done.  This is often something you have direct control over but sometimes is not.

Things includes your experiential base.  These are the issues you’ve seen before, the skills you have, the norms and values you’ve learned, and your general education and culture.  It includes the training related to your use of tools.

Things includes your environment.  An environment is a complex, composite concept.  It includes the people in that environment as well as their individual experiential bases and decisions.

I think that’s pretty well covered by people who have thought things out more than I have, but what I never see mentioned is that in the three surfaces of influence in a sphere is another dynamic called time.

Things move from a ‘direct control’ surface to an ‘influence-only’ surface all the time or from a “can’t influence” surface to a “direct control” and over and back.  The planes of these surfaces are not necessarily fixed.

For a more complex observation, while I tend to overuse the boyd loop as I find organic information processing cycles to be fascinating, I think it demonstrates how studied the concept of influence of typically uninfluenceable things is even by our military:

Granted the boyd loop was originally used by the USAF for combat operations ad-hoc strategies, but it was later repurposed for all kinds of operations in marketing and social activism or propaganda efforts by a vast array of larger organizations.  It’s widely known that you can influence things you should not believe that you can influence.

So, when someone references the serenity prayer to you make you feel better about having sucked at adulting, like so:

It’s comforting.  But you simply failed to influence your environment.  You have failed at adulting.

It happens to the best of us.

That said, some things would require an unrealistic amount of effort to influence.

As to coping with failure, I think Batman is the best metaphor.

If Bruce Wayne weren’t still useful to Batman he would just be Batman all the time.  Just because Batman’s life as Bruce Wayne is a lie doesn’t mean that Bruce Wayne wasn’t a useful character in the story — in fact it brought alot of balance to the Batman character — Bruce was the psychological success factor balancing Batman’s degrading psychological state.

In fact you could say that Bruce wasn’t really anything negative at all, he was just socially or emotionally discontiguous to the dominant persona telling the story.

Everybody needs a Bruce.  Everybody needs a Batman.  It’s just important to remember which persona is which when it comes to understanding characters and their decisions.

Examplar has reached about the 2/3rds completion mark.

/home/phanes/Development/internal/Examplar/cmake-build-debug/ftests
Parsed '/home/phanes/Development/internal/Examplar/conf/config.json' with 2 element(s).
Parsed '/home/phanes/Development/internal/Examplar/conf/units/all_test.units' with 1 element(s).
Added unit "independent test 1" to Suite.
Added unit "independent test 2" to Suite.
Added unit "dependent test" to Suite.
Parsed '/home/phanes/Development/internal/Examplar/conf/plans/test.plan' with 1 element(s).
Added task "independent test 1" to Plan.
Added task "independent test 2" to Plan.
Added dependency "independent test 1" to task "dependent test".
Added task "dependent test" to Plan.
Loaded definition "independent test 1" for task "independent test 1".
Loaded definition "independent test 2" for task "independent test 2".
Loaded definition "dependent test" for task "dependent test".
Ready to execute all tasks in Plan.

Process finished with exit code 0

Project Status

At this point I’m ready to start implementing the execution of tasks in the plan.

I’ll need to decide on the behaviour of execution.

Currently it executes the executable at units[i].target.  If that returns a non-0 or does not print what is in units[i].output to STDOUT, then it will check if units[i].rectify is set — if it is, then it will execute a heal attempt at filepath units[i].rectifier.

So far it’s a pretty concrete process.

There is a decision to make.  Should it execute target then rectifier and then target again to see if it passed, or should it assume rectifier heals the broken condition? I believe the former is the case:  It should alternate between target and rectifier until a 0 exit code with output printed to STDOUT.   However, I would prefer to avoid an infinite loop in cases that, for whatever reason a user’s rectifier script is not healing the condition that causes target to fail.

Without intervention, I will have it execute target, rectifier on fail, and then target again before checking units[i].required.  If that bool is set to true it should halt the entire plan.  If not, it should note it and move on.

This behaviour will be important later in determining how dependencies really work in Examplar.  If a dependency is listed in a Task, and that dependency failed but has a required attribute of false, then we’d be overriding that required attribute by checking for the completion flag of that failed dependency and simply halting.  This may be desired behaviour.  I am seeking feedback on this.

Data Integrity Assurance

There’s still some work to do with data integrity assurance during linking of tasks and units but it is currently working reliably when fed intact files.

Exceptions

The exceptions will need reworked eventually. They are accurate but somewhat lacking in references. Don’t want to do that to anyone who would end up supporting this. Luckily I’ve used strict OOP designs in this so it should be pretty easy to rework the exception later when the time comes.

Documentation

I’ve moved to Doxygen style documentation for all classes and methods.

Logging

We’re at a point where logging patterns should be considered.

Now that we’ve got most of the infra movement out of the way, or at least locked into a good plan for the next iteration of it, I am devoting all resources to the completion of Examplar to reach our first major milestone.

As a Generalist first, and historically a Python developer of sorts it has certainly been fun expanding my horizons to relearn C++ in a little more depth than I previously had a need to know.  Not only this, but working with data structures is just good for your brain.  Or bad for it depending on your perspective.

We’re Not Using Pagure

After further evaluation we realized that Pagure would not suit our needs.

We went through a gruelling install process only to find that it required service coupling that we weren’t comfortable with.  It was excellent working with the Pagure team during this process and I saw a deluge of commits as they head towards a new major release that hopefully will solve many of the things we encountered, but all in all it still is a great product and we wish them well.

We will instead be splitting what it would provide us between a refresh of gitlab and a new issue tracker referenced below.

New Official Source Code Repositories

I am pleased to announce that our new gitlab server is launched and looking spiffy.

Thanks to Greybeard Prawn for his diligence in security configuration to get us our A+ rating at Qualys SSL Labs.

The new URL for our source code repos is:

https://source.surroindustries.com

Registration is open, and encouraged to the public.

Please see an admin in irc://irc.freenode.org/#surro-linux if you have trouble with access.  We will be tuning access controls and default configurations for a while.

New Official Issue Tracker

I am also pleased to announce that our official bug tracker and request portal is launched.  The greybeards have been throwing rocks at it for a couple weeks to make sure it holds up.  This has made it possible for us to get alot more done in a shorter amount of time than we expected.

The new URL for our issue tracker is:

https://bugs.surroindustries.com

Registration is open, and encouraged to the public.

Please see an admin in irc://irc.freenode.org/#surro-linux if you have trouble with access.  We will be tuning access controls and default configurations for a while.

Next-Gen: New Salt Master

It is not able to be showcased yet but we’re currently in the progress of setting up a salt master for generation of a preprod environment.  Once this is complete we’ll generate a new prod environment from that.  After this is looking good we’ll cut over the DNS to the new prod.  Once this is in place we’ll be enjoying the benefits of infrastructure as code.

If you’d like to participate in this upcoming project please see Greybeard Prawn in irc://irc.freenode.org/#surro-linux.

Next-Gen: SSO/LDAP/Confederated User Management Systems

We are researching confederated user management for web services as all current services and systems have excellent LDAP integration.  I have nothing further to announce in this area right now.

Next-Gen: ISA

We’re also working on designing user/group and access control to lay groundworks for a support staff team member to join our ranks safely.  I have nothing further to announce in this area right now.

Pending: Blog Rotations

On a final announcement, we’ll be moving this blog to the SURRO umbrella domain along with pastebin services.  Currently my personal blog is at blog.surroindustries.com where I plan on making the new home for this blog.  My personal blog will be moving to the SILO umbrella at blog.silogroup.org.

Next-Gen: Mailing Lists

I am still deciding the role mailing lists will play in SURRO but I do think they’ll end up being important.  I need more time to think about this.

Open: Examplar

Now that infra is moving on its own a little I can focus on Examplar again when I next have time.

So, the last month or so has been extremely eventful in so many ways.

I don’t even know where to start.

SCM and Issue Trackers

I spent a few weeks evaluating various SCM systems and issue trackers for SURRO.  Ended up settling on mantisbt.  I’m actually really glad I found it as it was much better than what we’d initially set out for.

We were looking at potentially migrating gitlab but decided to stick with it as our SCM and migrate it to the SURRO domain.

I got to see a little deeper how to install gitlab from source.  Not for the feint of heart.  We’ll be using omnibus.

Interestingly enough once the issue tracker got up we had a huge surge of productivity that’s maintained ever since.  Just having it there made things more organized.

New Baby Project: checkinstall

Along the way found that checkinstall doesn’t compile on newer systems and is a dependency of gitlab build process.  Work journal is here.  I’ll be making it into an official patch and offering a patched fork on the SURRO repos.

Also found that our target system, Fedora 25 doesn’t offer omnibus packaging for gitlab, but CentOS does.  I suspect this is likely due to the fact that checkinstall from upstream doesn’t compile.

So once on the SURRO repos we’ll try to get a package included in the fedora repos.  I may need support from the checkinstall project owner to do this.

EMAIL

Had my yearly bout with postfix on the new prod buildout.  Still hate postfix.  We gotz emails.

SALT

Greybeard Prawn is working on salt master configuration to build a preprod with.  Once done he’ll be able to generate a new prod with it and we’ll cut over DNS.  Perfection.

Examplar vs. Support Infrastructure

No updates.  Between dayjob and my picking away at an infra push for SURRO I haven’t had time.  We’re laying groundworks down for staff to maintain all that once in place though so there’s that.  I need a vacation – I’m fighting burnout from spending too much time in front of a computer.

I’ve decided we’ll borrow from ITIL where appropriate but not for the purpose of implementing ITIL.

Greybeard DJ_L and Greybeard Son_Goku have been huge helps in offloading some of the work as it’s ramped up and I’m hoping we’ll reach a point where I can step away from infra and focus solely on Examplar and the broader SURRO Lifecycle.

SSO / Web Services User LifeCycle Management

Looking at FreeIPA + Ipsilon. Still evaluating.

Branding / Migrations / Server Refreshes

This shit is taking forever.  I’ll be moving this blog under the surro umbrella and the new official surro linux blog will either be in its place or at linux.*.  Don’t know yet.  Still looking at the diagram.  I was pretty sloppy when I built out the first round of stuff and now I’m paying for it with sweat and tears.  You wouldn’t believe how much planning goes into this.

SSL

We’re going to an HSTS-centered configuration for web services.  Stay tuned.

 

So, we’ve finally decided on a kickass system for new SCM services.  We’ve opted to go with Pagure.

Initially we’d evaluated quite a few systems and had decided on Apache Bloodhound.  Then, quite literally yesterday, we heard back from one of their lead devs that the project was dead.

So we needed something.  We’d evaluated Redmine.  No.  Bugzilla.  No.  Apache Allura.  Good lord, NO.  Trac.  If we have to but please no.

Then, Greybeard Son_Goku came to the rescue and suggested Pagure.

It had all the appeal bloodhound did only was a live project, and actually seemed to have better git integration than bloodhound did, so, I spent most of today going through their documentation and getting it installed.

Pagure installation required some debugging.  Its instructions provide two ways to install it.  One with setup.py and one using the rpm package manager.  I found that the setup.py instructions were broken, specifically as it pertained to the createdb script, which required me to alter the table structures in the database to get working and needs patched to be working, and also where we had to use a tool called alembic (which I will never mention in this blog again), as some setup is missing in the instructions.  The rpm path took care of both of those issues as long as we went with a postgresql database backend.

Except not, there was still quite a bit of deviation necessary to get it installed, and we still haven’t configured all of it, this was just to get it up and running.  I have created a work journal for pagure which you can observe below.  I will also have a separate journal for you to use when setting up your MTA. There is also yet another journal I will provide to integrate a pagure instance with a remote gitlab install since we are keeping gitlab for our source code repo hosting service.

@@2017-05-21
@1355 spoke to pingou
sts is primary dev for pagure
provided link:
	https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/pagure/frontend/tasks/main.yml
ansible script for installing pagure to use as reference for getting ours up

@1355 reimaged aether to Fedora 25

-- dark souls --

@1530 set up postgresql
$ sudo dnf install postgresql-server
$ sudo postgresql-setup --initdb
$ sudo systemctl enable postgresql
$ sudo service postgresql start
$ su - postgres
$ pgsql
$ -- set password for postgres superuser --

@1544 disabled selinux in /etc/selinux/config

@1600 -- enabled and started httpd --
$ systemctl enable httpd
$ service start httpd
$ iptables -P INPUT ACCEPT
$ iptables -F
$ firewall-cmd --zone=dmz --add-port=80/tcp
$ firewall-cmd --runtime-to-permanent
-- verified httpd is up --

@1611 verified from pingou redis is a required component
provided link:
	https://docs.pagure.org/pagure/overview.html

@1614 -- started install for pagure --
pingou suggests to use link below for rpms:
https://infrastructure.fedoraproject.org/repo/infra/ 
pagure is not in that repo
pingou advised to use rpmbuild -ta against a tarball from:
	 https://releases.pagure.org/pagure/
suggests 2.14.2 as stablest

@1623
$ adduser pagure
$ usermod -a -G wheel pagure
$ passwd pagure
$ su pagure
$ cd
$ wget https://releases.pagure.org/pagure/pagure-2.14.2.tar.gz
$ sudo dnf install rpm-build
$ rpmbuild -ta pagure-2.14.2.tar.gz
/*
error: Failed build dependencies:
	py-bcrypt is needed by pagure-2.14.2-1.fc25.noarch
	python-alembic is needed by pagure-2.14.2-1.fc25.noarch
	python-arrow is needed by pagure-2.14.2-1.fc25.noarch
	python-binaryornot is needed by pagure-2.14.2-1.fc25.noarch
	python-bleach is needed by pagure-2.14.2-1.fc25.noarch
	python-blinker is needed by pagure-2.14.2-1.fc25.noarch
	python-chardet is needed by pagure-2.14.2-1.fc25.noarch
	python-cryptography is needed by pagure-2.14.2-1.fc25.noarch
	python-docutils is needed by pagure-2.14.2-1.fc25.noarch
	python-enum34 is needed by pagure-2.14.2-1.fc25.noarch
	python-fedora is needed by pagure-2.14.2-1.fc25.noarch
	python-filelock is needed by pagure-2.14.2-1.fc25.noarch
	python-flask is needed by pagure-2.14.2-1.fc25.noarch
	python-flask-multistatic is needed by pagure-2.14.2-1.fc25.noarch
	python-flask-wtf is needed by pagure-2.14.2-1.fc25.noarch
	python-markdown is needed by pagure-2.14.2-1.fc25.noarch
	python-munch is needed by pagure-2.14.2-1.fc25.noarch
	python-nose is needed by pagure-2.14.2-1.fc25.noarch
	python-openid is needed by pagure-2.14.2-1.fc25.noarch
	python-openid-cla is needed by pagure-2.14.2-1.fc25.noarch
	python-openid-teams is needed by pagure-2.14.2-1.fc25.noarch
	python-psutil is needed by pagure-2.14.2-1.fc25.noarch
	python-pygit2 >= 0.20.1 is needed by pagure-2.14.2-1.fc25.noarch
	python-pygments is needed by pagure-2.14.2-1.fc25.noarch
	python-redis is needed by pagure-2.14.2-1.fc25.noarch
	python-sqlalchemy > 0.8 is needed by pagure-2.14.2-1.fc25.noarch
	python-straight-plugin is needed by pagure-2.14.2-1.fc25.noarch
	python-wtforms is needed by pagure-2.14.2-1.fc25.noarch
	python2-devel is needed by pagure-2.14.2-1.fc25.noarch
	systemd-devel is needed by pagure-2.14.2-1.fc25.noarch
*/
$ sudo dnf install $(rpmbuild -ta pagure-2.14.2.tar.gz 2>&1 | awk '{ print $1; }' | grep -v "error" | sort -n | tr '\n')
$ rpmbuild -ta pagure-2.14.2.tar.gz
$ sudo dnf install /home/pagure/rpmbuild/RPMS/noarch/pagure-*.rpm
$ sudo mkdir -p /var/www/releases
$ sudo chown pagure:pagure /var/www/releases
$ sudo mkdir -p /srv/git/repositories/{docs,forks,tickets,requests,remotes}

-- configured apache's pagure.conf --

$ cat /etc/httpd/conf.d/pagure.conf 
/*
#WSGISocketPrefix run/wsgi
##WSGIRestrictStdout On
#WSGIRestrictSignal Off
#WSGIPythonOptimize 1
#WSGIPassAuthorization On
#WSGIDaemonProcess pagure user=git group=git maximum-requests=1000 display-name=pagure processes=4 threads=4 inactivity-timeout=300
## It is important that the doc server runs in a different apache process
#WSGIDaemonProcess paguredocs user=git group=git maximum-requests=1000 display-name=pagure processes=4 threads=4 inactivity-timeout=300

#<VirtualHost *:80>
  #ServerName pagure.io
  #Redirect permanent / https://pagure.io/
#</VirtualHost>


#<VirtualHost *:80>
  #ServerName docs.pagure.org
  #Redirect permanent / https://docs.pagure.org/
#</VirtualHost>


#<VirtualHost *:443>
  #ServerName docs.pagure.org

  #WSGIScriptAlias / /usr/share/pagure/docs_pagure.wsgi

  #SSLEngine on
  #SSLProtocol all -SSLv2 -SSLv3
  ## Use secure TLSv1.1 and TLSv1.2 ciphers
  #Header always add Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"

  #SSLCertificateFile /etc/pki/tls/....crt
  #SSLCertificateChainFile /etc/pki/tls/....intermediate.crt
  #SSLCertificateKeyFile /etc/pki/tls/....key

  #Alias /static /usr/lib/python2.7/site-packages/pagure/static/

  #<Location />
    #WSGIProcessGroup paguredocs
    #<IfModule mod_authz_core.c>
      ## Apache 2.4
      #Require all granted
    #</IfModule>
    #<IfModule !mod_authz_core.c>
      ## Apache 2.2
      #Order deny,allow
      #Allow from all
    #</IfModule>
   #</Location>
#</VirtualHost>


#<VirtualHost *:443>
  #ServerName pagure.io

  #WSGIScriptAlias / /usr/share/pagure/pagure.wsgi

  #SSLEngine on
  #SSLProtocol all -SSLv2 -SSLv3
  ## Use secure TLSv1.1 and TLSv1.2 ciphers
  #Header always add Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"

  #SSLCertificateFile /etc/pki/tls/....crt
  #SSLCertificateChainFile /etc/pki/tls/....intermediate.crt
  #SSLCertificateKeyFile /etc/pki/tls/....key

  #Alias /static /usr/lib/python2.7/site-packages/pagure/static/
  #Alias /releases /var/www/releases

  ## Section used to support cloning git repo over http (https in this case)
  #SetEnv GIT_PROJECT_ROOT /srv/git/repositories

  #AliasMatch ^/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$          /path/to/git/repositories/$1
  #AliasMatch ^/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /path/to/git/repositories/$1
  #ScriptAliasMatch \
    #"(?x)^/(.*/(HEAD | \
    #info/refs | \
    #objects/info/[^/]+ | \
    #git-(upload|receive)-pack))$" \
    #/usr/libexec/git-core/git-http-backend/$1

  #<Location />
   #WSGIProcessGroup pagure
   #<IfModule mod_authz_core.c>
      ## Apache 2.4
      #Require all granted
   #</IfModule>
   #<IfModule !mod_authz_core.c>
      ## Apache 2.2
      #Order deny,allow
      #Allow from all
   #</IfModule>
  #</Location>

  ## Folder where are stored the tarball of the releases
  #<Location /releases>
   #WSGIProcessGroup pagure
   #<IfModule mod_authz_core.c>
      ## Apache 2.4
      #Require all granted
   #</IfModule>
   #<IfModule !mod_authz_core.c>
       ## Apache 2.2
      #Order deny,allow
      #Allow from all
   #</IfModule>
  #</Location>

  #<Directory /var/www/releases>
    #Options +Indexes
  #</Directory>

#</VirtualHost>
*/

$ sudo vim /etc/httpd/conf.d/pagure.conf
$ cat /etc/httpd/conf.d/pagure.conf 
/*
WSGISocketPrefix run/wsgi
WSGIRestrictStdout On
WSGIRestrictSignal Off
WSGIPythonOptimize 1
WSGIPassAuthorization On
WSGIDaemonProcess pagure user=pagure group=pagure maximum-requests=1000 display-name=pagure processes=4 threads=4 inactivity-timeout=300
## It is important that the doc server runs in a different apache process
WSGIDaemonProcess paguredocs user=pagure group=pagure maximum-requests=1000 display-name=pagure processes=4 threads=4 inactivity-timeout=300

#<VirtualHost *:80>
  #ServerName pagure.io
  #Redirect permanent / https://pagure.io/
#</VirtualHost>


#<VirtualHost *:80>
  #ServerName docs.pagure.org
  #Redirect permanent / https://docs.pagure.org/
#</VirtualHost>


<VirtualHost *:80>
  ServerName scm-docs.surroindustries.com

  WSGIScriptAlias / /usr/share/pagure/docs_pagure.wsgi

  # SSLEngine on
  # SSLProtocol all -SSLv2 -SSLv3
  ## Use secure TLSv1.1 and TLSv1.2 ciphers
  #Header always add Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"

  #SSLCertificateFile /etc/pki/tls/....crt
  #SSLCertificateChainFile /etc/pki/tls/....intermediate.crt
  #SSLCertificateKeyFile /etc/pki/tls/....key

  Alias /static /usr/lib/python2.7/site-packages/pagure/static/

  <Location />
    WSGIProcessGroup paguredocs
    <IfModule mod_authz_core.c>
      ## Apache 2.4
      Require all granted
    </IfModule>
    #<IfModule !mod_authz_core.c>
      ## Apache 2.2
      #Order deny,allow
      #Allow from all
    #</IfModule>
   </Location>
</VirtualHost>


<VirtualHost *:80>
  ServerName scm.surroindustries.com

  WSGIScriptAlias / /usr/share/pagure/pagure.wsgi

  #SSLEngine on
  #SSLProtocol all -SSLv2 -SSLv3
  ## Use secure TLSv1.1 and TLSv1.2 ciphers
  #Header always add Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"

  #SSLCertificateFile /etc/pki/tls/....crt
  #SSLCertificateChainFile /etc/pki/tls/....intermediate.crt
  #SSLCertificateKeyFile /etc/pki/tls/....key

  Alias /static /usr/lib/python2.7/site-packages/pagure/static/
  Alias /releases /var/www/releases

  ## Section used to support cloning git repo over http (https in this case)
  SetEnv GIT_PROJECT_ROOT /srv/git/repositories

  AliasMatch ^/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$          /srv/git/repositories/$1
  AliasMatch ^/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /src/git/repositories/$1
  ScriptAliasMatch \
    "(?x)^/(.*/(HEAD | \
    info/refs | \
    objects/info/[^/]+ | \
    git-(upload|receive)-pack))$" \
    /usr/libexec/git-core/git-http-backend/$1

  <Location />
   WSGIProcessGroup pagure
   <IfModule mod_authz_core.c>
      ## Apache 2.4
      Require all granted
   </IfModule>
   #<IfModule !mod_authz_core.c>
      ## Apache 2.2
      #Order deny,allow
      #Allow from all
   #</IfModule>
  </Location>

  ## Folder where are stored the tarball of the releases
  <Location /releases>
   WSGIProcessGroup pagure
   <IfModule mod_authz_core.c>
      ## Apache 2.4
      Require all granted
   </IfModule>
   #<IfModule !mod_authz_core.c>
       ## Apache 2.2
      #Order deny,allow
      #Allow from all
   #</IfModule>
  </Location>

  <Directory /var/www/releases>
    Options +Indexes
  </Directory>

</VirtualHost>
*/

@1705 -- gave apache user permissions to git repositories --
$ setfacl -m user:apache:rx --default
/*
Usage: setfacl [-bkndRLP] { -m|-M|-x|-X ... } file ...
Try `setfacl --help' for more information.
*/

$ sudo setfacl -Rdm user:apache:rx /srv/git
$ sudo setfacl -Rm user:apache:rx /srv/git

@1713
$ sudo vim /etc/pagure/pagure.cfg 
$ cat /etc/pagure/pagure.cfg 
/*
import os
from datetime import timedelta

### Set the time after which the admin session expires
# There are two sessions on pagure, login that holds for 31 days and
# the session defined here after which an user has to re-login.
# This session is used when accessing all administrative parts of pagure
# (ie: changing a project's or a user's settings)
ADMIN_SESSION_LIFETIME = timedelta(minutes=20)

### Secret key for the Flask application
SECRET_KEY='<The web application secret key>'

### url to the database server:
#DB_URL=mysql://user:pass@host/db_name
#DB_URL=postgres://user:pass@host/db_name
DB_URL = 'postgres://****:****@localhost/pagure'

### The FAS group in which the admin of pagure are
ADMIN_GROUP = ['sysadmin-main']

### Hard-coded list of global admins
PAGURE_ADMIN_USERS = []

### The email address to which the flask.log will send the errors (tracebacks)
EMAIL_ERROR = 'errors@scm.surroindustries.com'

### SMTP settings
SMTP_SERVER = 'localhost'
SMTP_PORT = 25
SMTP_SSL = False

#Specify both for enabling SMTP with auth
SMTP_USERNAME = None
SMTP_PASSWORD = None

### Information used to sent notifications
FROM_EMAIL = 'pagure@scm.surroindustries.com'
DOMAIN_EMAIL_NOTIFICATIONS = 'scm.surroindustries.com'
SALT_EMAIL = '<secret key to be changed>'

### The URL at which the project is available.
APP_URL = 'https://scm.surroindustries.com/'
### The URL at which the documentation of projects will be available
## This should be in a different domain to avoid XSS issues since we want
## to allow raw html to be displayed (different domain, ie not a sub-domain).
DOC_APP_URL = 'https://scm-docs.surroindustries.com'

### The URL to use to clone git repositories.
GIT_URL_SSH = 'ssh://git@git.surroindustries.com/'
GIT_URL_GIT = 'git://git@git.surroindustries.com/'

### Folder containing to the git repos
GIT_FOLDER = os.path.join(
    os.path.abspath(os.path.dirname(__file__)),
    '..',
    'repos'
)

### Folder containing the docs repos
DOCS_FOLDER = os.path.join(
    os.path.abspath(os.path.dirname(__file__)),
    '..',
    'docs'
)

### Folder containing the tickets repos
TICKETS_FOLDER = os.path.join(
    os.path.abspath(os.path.dirname(__file__)),
    '..',
    'tickets'
)

### Folder containing the pull-requests repos
REQUESTS_FOLDER = os.path.join(
    os.path.abspath(os.path.dirname(__file__)),
    '..',
    'requests'
)

### Folder containing the clones for the remote pull-requests
REMOTE_GIT_FOLDER = os.path.join(
    os.path.abspath(os.path.dirname(__file__)),
    '..',
    'remotes'
)

### Whether to enable scanning for viruses in attachments
VIRUS_SCAN_ATTACHMENTS = False

### Configuration file for gitolite
GITOLITE_CONFIG = os.path.join(
    os.path.abspath(os.path.dirname(__file__)),
    '..',
    'gitolite.conf'
)

### Home folder of the gitolite user
### Folder where to run gl-compile-conf from
GITOLITE_HOME = None

### Version of gitolite used: 2 or 3?
GITOLITE_VERSION = 3

### Folder containing all the public ssh keys for gitolite
GITOLITE_KEYDIR = None

### Path to the gitolite.rc file
GL_RC = None

### Path to the /bin directory where the gitolite tools can be found
GL_BINDIR = None

# SSH Information

### The ssh certificates of the git server to be provided to the user
### /!\ format is important
# SSH_KEYS = {'RSA': {'fingerprint': '<foo>', 'pubkey': '<bar>'}}

# Optional configuration

### Number of items displayed per page
# Used when listing items
ITEM_PER_PAGE = 50

### Maximum size of the uploaded content
# Used to limit the size of file attached to a ticket for example
MAX_CONTENT_LENGTH = 4 * 1024 * 1024  # 4 megabytes

### Lenght for short commits ids or file hex
SHORT_LENGTH = 6

### List of blacklisted project names that can conflicts for pagure's URLs
### or other
BLACKLISTED_PROJECTS = [
    'static', 'pv', 'releases', 'new', 'api', 'settings',
    'logout', 'login', 'users', 'groups', 'projects']

### IP addresses allowed to access the internal endpoints
### These endpoints are used by the milter and are security sensitive, thus
### the IP filter
IP_ALLOWED_INTERNAL = ['127.0.0.1', 'localhost', '::1']

### EventSource/Web-Hook/Redis configuration
# The eventsource integration is what allows pagure to refresh the content
# on your page when someone else comments on the ticket (and this without
# asking you to reload the page.
# By default it is off, ie: EVENTSOURCE_SOURCE is None, to turn it on, specify
# here what the URL of the eventsource server is, for example:
# https://ev.pagure.io or https://pagure.io:8080 or whatever you are using
# (Note: the urls sent to it start with a '/' so no need to add one yourself)
EVENTSOURCE_SOURCE = None
# Port where the event source server is running (maybe be the same port
# as the one specified in EVENTSOURCE_SOURCE or a different one if you
# have something running in front of the server such as apache or stunnel).
EVENTSOURCE_PORT = 8080
# If this port is specified, the event source server will run another server
# at this port and will provide information about the number of active
# connections running on the first (main) event source server
#EV_STATS_PORT = 8888
# Web-hook can be turned on or off allowing using them for notifications, or
# not.
WEBHOOK = False

### Redis configuration
# A redis server is required for both the Event-Source server or the web-hook
# server.
REDIS_HOST = '0.0.0.0'
REDIS_PORT = 6379
REDIS_DB = 0

# Authentication related configuration option

### Switch the authentication method
# Specify which authentication method to use, defaults to `fas` can be or
# `local`
# Default: ``fas``.
PAGURE_AUTH = 'fas'

# When this is set to True, the session cookie will only be returned to the
# server via ssl (https). If you connect to the server via plain http, the
# cookie will not be sent. This prevents sniffing of the cookie contents.
# This may be set to False when testing your application but should always
# be set to True in production.
# Default: ``True``.
SESSION_COOKIE_SECURE = False

# The name of the cookie used to store the session id.
# Default: ``.pagure``.
SESSION_COOKIE_NAME = 'pagure'

# Boolean specifying whether to check the user's IP address when retrieving
# its session. This make things more secure (thus is on by default) but
# under certain setup it might not work (for example is there are proxies
# in front of the application).
CHECK_SESSION_IP = True

# Used by SESSION_COOKIE_PATH
APPLICATION_ROOT = '/'

# Allow the backward compatiblity endpoints for the old URLs schema to
# see the commits of a repo. This is only interesting if you pagure instance
# was running since before version 1.3 and if you care about backward
# compatibility in your URLs.
OLD_VIEW_COMMIT_ENABLED = False
*/

@1726 -- created the pgsql database --
$ PAGURE_CONFIG=/etc/pagure/pagure.cfg python /usr/share/pagure/pagure_createdb.py
/*
[pagure@scm noarch]$ PAGURE_CONFIG=/etc/pagure/pagure.cfg python /usr/share/pagure/pagure_createdb.py
Traceback (most recent call last):
  File "/usr/share/pagure/pagure_createdb.py", line 7, in <module>
    from pagure import APP
  File "/usr/lib/python2.7/site-packages/pagure/__init__.py", line 106, in <module>
    from flask_fas_openid import FAS
ImportError: No module named flask_fas_openid
*/

We are missing python-flask-openid in instructions and system.

$ sudo dnf install python-flask-openid
$ sudo dnf install python-fedora-flask
$ PAGURE_CONFIG=/etc/pagure/pagure.cfg python /usr/share/pagure/pagure_createdb.py
/*
Traceback (most recent call last):
  File "/usr/share/pagure/pagure_createdb.py", line 7, in <module>
    from pagure import APP
  File "/usr/lib/python2.7/site-packages/pagure/__init__.py", line 185, in <module>
    SESSION = pagure.lib.create_session(APP.config['DB_URL'])
  File "/usr/lib/python2.7/site-packages/pagure/lib/__init__.py", line 112, in create_session
    client_encoding='utf8')
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 386, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 75, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 579, in dbapi
    import psycopg2
ImportError: No module named psycopg2
*/

$ sudo dnf install python2-psycopg2
$ PAGURE_CONFIG=/etc/pagure/pagure.cfg python /usr/share/pagure/pagure_createdb.py
/*
Traceback (most recent call last):
  File "/usr/share/pagure/pagure_createdb.py", line 14, in <module>
    debug=True)
  File "/usr/lib/python2.7/site-packages/pagure/lib/model.py", line 74, in create_tables
    BASE.metadata.create_all(engine)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3762, in create_all
    tables=tables)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1855, in _run_visitor
    with self._optional_conn_ctx_manager(connection) as conn:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1848, in _optional_conn_ctx_manager
    with self.contextual_connect() as conn:
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
    e, dialect, self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
    return fn()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
    rec = pool._do_get()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get
    self._dec_overflow()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get
    return self._create_connection()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
    self.connection = self.__connect()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
    connection = self.__pool._invoke_creator(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  Ident authentication failed for user "pagure"
*/

@1828 -- set postgresql to allow password logins --
/var/lib/pgsql/data/pg_hba.conf needs altered for this to work.  
# local all all peer -> local all all password
$ sudo service postgresql restart

@1829 -- set /etc/pagure/pagure.cfg to use postgres name and pw --

$ PAGURE_CONFIG=/etc/pagure/pagure.cfg python /usr/share/pagure/pagure_createdb.py
/* same error as previous */

@1830 puiterwijk advised to use --username= and --password= to createdb script
/* same error as previous */

Added to pg_hba.conf:
# host all all 127.0.0.1/128 password
$ sudo service postgresql restart 

$ PAGURE_CONFIG=/etc/pagure/pagure.cfg python /usr/share/pagure/pagure_createdb.py
/*
Traceback (most recent call last):
  File "/usr/share/pagure/pagure_createdb.py", line 14, in <module>
    debug=True)
  File "/usr/lib/python2.7/site-packages/pagure/lib/model.py", line 74, in create_tables
    BASE.metadata.create_all(engine)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3762, in create_all
    tables=tables)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1855, in _run_visitor
    with self._optional_conn_ctx_manager(connection) as conn:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1848, in _optional_conn_ctx_manager
    with self.contextual_connect() as conn:
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
    e, dialect, self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
    return fn()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
    rec = pool._do_get()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get
    self._dec_overflow()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get
    return self._create_connection()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
    self.connection = self.__connect()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
    connection = self.__pool._invoke_creator(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  database "pagure" does not exist
*/

Our "createdb" script wants you to manually create the database before it will run.

@1857 -- create database pagure with user pagure --
$ su - postgres
$ psql -U postgres
$ >CREATE USER pagure WITH PASSWORD '****';
$ >CREATE DATABASE pagure;
$ >GRANT ALL PRIVILEGES ON DATABASE pagure to pagure;
$ >\q
$ PAGURE_CONFIG=/etc/pagure/pagure.cfg python /usr/share/pagure/pagure_createdb.py

-- success... --

@1904 -- do weird things with alembic that i really dont care about and shouldnt be needing to mess with --
$ sudo vim /etc/pagure/alembic.ini
	edit DB_URL uri (it's a URI)

$ alembic stamp $(alembic heads |awk '{ print $1 }')
/*
Using configuration file `/etc/pagure/pagure.cfg`
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 3ffec872dfdf
*/

ITS INSTALLED!

$ sudo service httpd restart

@1924 provided work log to pagure team for updates to pagure documentation

@1925 observed scm.surroindustries.com wants to use FAS login by default
puiterwijk on IRC advised PAGURE_AUTH = 'local' is what's needed for expected behaviour
$ sudo vim /etc/pagure/pagure.cfg
# PAGURE_AUTH = 'fas' -> PAGURE_AUTH = 'local

Completed.

@1935 -- realized the MTA is not setup and new user registration will produce a 500 error if not set up --
@1945 -- realized no one in the entire project will set up the MTA for me --

--Chris Punches

What is Surro Linux?

• Surro Linux is an organization of contributors intending to build a fully LSB 5.0 certified and FHS 3.0 compliant Linux distribution.  It will use RPM for its package manager and will have public repositories.
• Surro Linux is not based on another distribution and is not a “spin” of an existing Linux Distribution.
• It is not embedded Linux.

What are notable milestones to get there?

While it takes about a day to create something that will just boot, we’ve found that an enormous amount of the work is in designing it to be usable on a large scale by others in a consistent fashion (setting up infrastructure, source code management tools, designing how it will be version controlled and the filesystem structures to use, designing the actual system to be used beyond just a POC, all of our support services for the design team, there are a million things that make releasing a real distro different than just building an embedded linux system (which some people confuse this effort to be sometimes), and many of those “extra things” that come with a real release we’re examining how best to do as an organization still.  If we just wanted a bootable disc, we would have already released it.  Getting to a boot is easy, until you add even a single other user, let alone LSB 5 compliance, which no other distro has done yet.

So, to understand the milestones, you’ve got to understand that this is not just one project, but a project that is a collection of projects:  Examplar, Foster, Surro Linux.

Milestone 1:  Build Examplar.

Examplar is a test and automation framework built for Surro Linux.  It is designed to be a generic automation tool and is nearing completion.  I am currently building this myself while building out infrastructure and I am taking my time.

Milestone 2:  Use Examplar to Build Foster

Once Examplar is complete enough, it will be used to execute the build scripts from our previous iterations (or something kind of like them) to build the boot environment.  This environment is a bootable ISO called Foster.

Foster is used to build the actual distribution.

Using Foster instead of cross compiling everything in a weird kind of soup dump will save us an ungodly amount of time troubleshooting the scripts being executed and this also ensures a sane build environment.

Milestone 3: Use Foster to build the SURRO prototype image.

Due to the nature of Examplar’s design it ought to be able to drive the booted system using a differently configured version of itself on the build iso that will proceed to build out the actual distribution image using packages from our public repository and black magic.

At this phase users may be able to demo the image but installation will be ugly.

Milestone 4:  Build an installer that puts SURRO on a local disc

At this phase we’ll be putting efforts into stripping the prototype image down to just enough to build an installer on it and then building that installer.  It’ll likely be curses-driven for a while.

Milestone 5:  Creating a lifecycle — Self Maintaining Process.

At this phase we’ll be shifting efforts into creating a lifecycle for every aspect of Surro Linux.  This includes package version management and generation (we’re looking at mock for package generation and if you’re familiar with CI/CD we’ll probably be using some workflows you recognize).  It includes sustainable design process design (that’s a mouthful, and also a thing).

Every milestone prior to this will need to be done correctly for Milestone 5 to be done well, so, Milestone 5 should be thought of as being split among all the other milestones and is a huge timesink but is worth it.  The intention is to design it to be maintainable by a small crew, able to accommodate a large crew, and keep itself going once we make a release.

Timeline:

As it gets completed.  It is estimated to be a 2 year project to Alpha release.  We’re taking our time with it and if you’d like to contribute we are definitely looking for more people to speed it up.

Who is Behind Surro?

Surro is being designed by what our org refers to as “Greybeards”.  These are contributors from large projects, small projects, and no projects.  Some are contributors to the Linux From Scratch project, some are contributors to various Fedora projects (Fedora also has subprojects like we’ve outlined here).  Some are from smaller but equally notable projects.  And then there’s me.  I’m Phanes.  I’m from nowhere.  I do things.

Associated Links:

linux.silogroup.org

This journal.  Tracks updates and decisions for stakeholders.  In migration to serve as high level summary to decisions better discussed in a ticketing system that will also be referenced by some of our git repos.

git.surroindustries.com

Our git repository host.  Hosts git repositories with the exception of Examplar which is on Github currently until our SCM infrastructure migration is complete and is not intended for general use by the public but is open registration.

www.silogroup.org

Landing page for the surro design org’s parent organization, SILO GROUP.  They’re gonna make the Linux Great Again.  They’re basically just a holding umbrella for projects.

blog.surroindustries.com

My offtopic feed for updates.  Mostly where I spew and whine and complain about unrelated things but sometimes related things.  Sometimes passive aggressively.

repo.luxidolon.com

Our currently pathetic source code repository to be used by some of our build scripts.  Subject to radical structure changes, considered unstable, and will likely be moved.  There are talks of edu domain hosting sponsorship taking place.

List of Additional Hoops We Should Have to Jump Through For This:

1.

Question, Comments, Concerns:

Our design team resides on irc.freenode.org/#surro-linux and we have limited presence on irc.oftc.net/#surro-linux.

Home is where your heart is …buried

After heated negotiations, 21 duels, a nuclear arms race and several strategically arranged noble marriages, I’m excited to announce that we’ve relocated back to freenode for our official channel.  We’re already seeing the pace pick back up.

irc://irc.freenode.org/#surro-linux

We’re currently looking for C++ devs.  Stop on by.

New Infrastructure

I’ve been evaluating various SCM components to bring in in lieu of a mailing list and think we’re pretty close to making a decision.  More updates to follow in a few days.

Server Refresh

We’re going to be doing a server refresh soon to defuzz branding later.  Whatever that means.

 

I had forgotten how slowly C++ development moves when your specialization is not in C++ development (and mine totally is not — actually, what is mine?).

At this point I’m bundling a small json parsing implementation with Examplar to avoid creating a runtime dependency.

There really ought not be any other library inclusion needs besides some simple subprocess spawning logic for triggering a task‘s unit‘s target executable and healer executable, as well as grabbing the expected output from the execution of targets/healers specified in the test unit file which has a path defined in config.json.

If that’s a little compact, this may make more sense:

===========================================================================
Output
===========================================================================
/home/phanes/Development/internal/ftests/cmake-build-debug/ftests
---------------------------------------------------------------------------
Parsed config.json with 2 elements.
Parsed ./conf/units/all_test.units with 1 elements.
Parsed ./conf/test.plan with 1 elements.
Found task name in "./conf/test.plan":	gcc is present

Associated Unit name:		gcc is present
Associated Unit target:		./ubuntu/xenial/check_gcc_present.run
Associated Unit healer:		./ubuntu/xenial/install_gcc
Associated Unit heals:		true

Found task name in "./conf/test.plan":	gcc can compile

Associated Unit name:		gcc can compile
Associated Unit target:		./ubuntu/xenial/check_gcc_compiles.run
Associated Unit healer:		echo pass
Associated Unit heals:		false


Process finished with exit code 0

===========================================================================
Input
===========================================================================
---------------------------------------------------------------------------
config.json
---------------------------------------------------------------------------
{
  "units_path":	"./conf/units/all_test.units",
  "plan_path":  "./conf/test.plan"
}

---------------------------------------------------------------------------
./conf/units/all_test.units
---------------------------------------------------------------------------
{
    "units": [
        {
            "name":             "gcc is present",
            "target":           "./ubuntu/xenial/check_gcc_present.run",
            "output":           "present",
            "rectifier":	    "./ubuntu/xenial/install_gcc",
            "active":	        true,
            "required":         true,
            "rectify":          true
        },
        {
            "name":             "gcc can compile",
            "target":           "./ubuntu/xenial/check_gcc_compiles.run",
            "output":           "can compile",
            "rectifier":	    "echo pass",
            "active":	        true,
            "required":         true,
            "rectify":          false
        }
    ]
}

---------------------------------------------------------------------------
./conf/test.plan
---------------------------------------------------------------------------
{
    "plan": [
        { "name": "gcc is present", "depends on":  [ null ] },
        { "name": "gcc can compile", "depends on":  [ "gcc is present" ] }
    ]
}

Once Examplar is complete, SURRO will be able to be built modularly to accommodate profiles, which are just a set of tasks (a plan) and units (what the plan executes) provided to run in different environments and will be community contributions.

I am delighted by this as it means other types of projects which have nothing to do with SURRO or Foster will be quite well facilitated by the design.

One thing that will need to be done (in a future iteration, I do not need this feature myself yet), is to have units_path in config.json be a directory path full of json files defining their units that are then amalgamated to produce a cascading style of configuration for the user.

While I do not need that or want to use that myself, seriously, I do think that this will make breaking unit definitions into sets, like desktop_packages.unit and base_system.unit will be useful later for building Foster as that project evolves (so long as I stop creating needs for tools that don’t exist yet).

So from here is to polish up the encapsulation needs, create a subprocess handling set of classes to manage the flow in a rigid manner according to plan, and from there we’ll want to give it:

• A default path set for config.json.
• Cascading unit definitions (eventually).
• A unix-style command-line interface (a simple one).

On the PR front, I’ve got exciting news that I can’t share yet.

After examining the python-bootstrap and build system for foster that we’ve been working on, in the context of the previous post about a potential port to C being in analysis, I’ve found a way to do all of this in a much simpler way.

First, a re-examining of the requirements.

We need a test system that is:

• for environment variables, package or binary versions, package or binary presence, symlink targets, and all manners of other little flexible checks that could really be looking at anything and be done in different ways on different linux systems.
• relatively abstracted away from the distribution running it without having to alter the implementation of the test system every time we find a new type of check.
• Massively extendable in a simple way that can either result in atomic use cases that are easy to implement or absolutely gigantic and comprehensive uses across many purposes for more robust solutions.
• Designed in a way that prevents configuration complexity from scaling with usage complexity (huge solutions are just as easy to configure as small solutions).
• Rigidly simple in testing order.

Introducing: Examplar

“Not your father’s testing framework.”

First, some updates to the data model allowed the new design to take shape:

{
  "units":	[
    {
      "name":             "bash",
      "target":           "./ubuntu/xenial/check_bash_version.run",
      "output":           "4.3.46(1)-release",
      "rectifier":	      "./ubuntu/xenial/install_correct_bash.run",
      "active":	          true,
      "required":         true,
      "rectify":          false
    },
    {
      "name":             "bash2",
      "target":           "./ubuntu/xenial/check_bash_version.run",
      "output":           "4.3.46(1)-release",
      "rectifier":	      "./ubuntu/xenial/install_correct_bash.run",
      "active":	          true,
      "required":         true,
      "rectify":          false
    }
  ],
  "plan": [
    { "name": "bash", "depends on":  [ null ] },
    { "name": "bash2", "depends on":  [ "bash" ] }
  ]
}

To read the above, we have two JSON objects here.  One is units, which define/declare the tests themselves.  The other is plan which orders the execution of units to be tested and sets test interdependency (flow control, execution, implementation).

The tool will cycle through the plan object’s entries, executing units using their unique name attribute as a handle.

To do this, the tool will deserialize units, execute each corresponding unit’s target value when that unit’s active property is true and is next in the plan object’s test order.  The unit’s target attribute points to an executable.  That executable serves as the test.  If target returns a non-zero exit code or if target‘s output does not match the unit’s output attribute value then the test will fail.  If the unit’s required attribute is set to true it will stop the tests.  Provided these all happen well it will continue to the next unit test in the deserialized collection by order of what’s in plan.

If a test is dependent on other assumptions, say, in our use case we have “making sure GCC compiles” we want that to depend on “making sure GCC is present”, we’d have something like:

{
  "units":	[
    {
      "name":             "gcc is present",
      "target":           "./ubuntu/xenial/check_gcc_present.run",
      "output":           "present",
      "rectifier":	      "./ubuntu/xenial/install_gcc",
      "active":	          true,
      "required":         true,
      "rectify":          true
    },
    {
      "name":             "gcc can compile",
      "target":           "./ubuntu/xenial/check_gcc_compiles.run",
      "output":           "can compile",
      "rectifier":	      "",
      "active":	          true,
      "required":         true,
      "rectify":          false
    }
  ],
  "plan": [
    { "name": "gcc is present", "depends on":  [ null ] },
    { "name": "gcc can compile", "depends on":  [ "gcc is present" ] }
  ]
}

 

And what will happen is this:

• This tool will load plan
• locate a unit called “gcc is present” first from units
• check if units["gcc is present"].active is true
• execute that unit’s target value as a subprocess
• fail on either non-zero exit of target or if target‘s output does not match that unit’s outputvalue (checks for “present” in stdout to subprocess call of ./ubuntu/xenial/check_gcc_present.run
• if it failed, execute "gcc is present".rectifier which is an executable that is intended to repair the failed condition of the test.
• It will then loop between the rectifier and target to allow the creation of multi-pass repairs for obscure conditions (like race conditions)
• When "gcc is present".target finally returns 0 and echos “present” to STDOUT, the tool will print the pretty systemD style output of [ PASS ] "gcc is present" and move to the next unit in plan, which is “gcc can compile”.  If units["gcc can compile"].active is true ….
• It will then run the script that attempts a gcc compile based on units["gcc can compile"].target, which should return 0 and echo “can compile” if all goes well.

Obviously the design was intended to accomodate even very, very complex use cases with quite straightforward configuration this way and you can even do some metascripting with this (though I won’t and I don’t know why you would).

The solution language is currently C++ since that’s going to be a pretty safe dependency assumption on the builder’s system (and since not having G++ installed will cause the tests we’re using in our use case to fail anyway).

Great.  Why do I care?

This design will be able to build all of Foster.  And not just that — it’ll force the linear nature of these builds to occur in proper order with limited assumptions about environment and toolchains — and without the pain of trying to enforce a consistent scripting pattern through what was turning into dozens of shell scripts or introducing heavy dependencies, while still being abstract enough to allow the introduction of mixed-language solutions for the gnarly bits.  It accomodates profiles for any distribution, and is reusable for things outside of foster — if you check out SURRO Industries’ personal blog it’ll even be able to serve as the engine for the Conditional Response Daemon (CRD) that was put on the backburner for SURRO Linux and Surro Linux’s Foster initiative.

That’s all for you though.  Why do *I* care?

This will be useful for unit tests of one’s codebase by QA Engineers in software engineering environments, Infrastructure Engineers validating the requirements are met of a server build operation, Build and Deployment Engineers (I’m thinking DevOps engineers) to build and deploy or who are ensuring that their deployment job or build job generated all the files they want and that those files ended up on the right servers — the list goes on, and on and on for what this can be used for as long as I keep the design abstracted from Foster and SURRO as a generic tool.  The target executables can even include curses menus or flow control prompts for extremely elegant and potentially very complex operations that could be designed by one party or department to be executed by another party or department (process design in your org now has a tool that keeps it separated from process implementation — this means you, the entrepreneur can tell another entity or department how to do things at the high level without confusion over what exact change was made during your RCA during a failed rollback or any of the millions of situations that can come up in an SDLC).

I hope that others find this useful, and consider donating to the SURRO Linux Development Fund to sponsor more development hours as designs for these types of tools are innumerable in the talks coming out of the new official greybeards channel on irc://irc.oftc.net/#surro-linux.

So when do I get to use it?

As soon as:

1. I complete it, which is required as a next step to the Foster design and build out.
2. It has an official name.  There are candidates waiting on the rest of the Council of Greybeards:

• Marbles
• Chauffeur
• Examplar
• Blinkers
• Pebbles
• Mane

It is requested of any Greybeards reading this to submit your selection of a name to the channel over the next 24 hours.  The default name is Examplar unless we have consensus on another name.

Note:

After discussion with DJ the units will be able to be placed in multiple files (or one big file) to be read in a cascading fashion from a directory referenced in a config.ini (think puppet manifests or apache.conf).  The plan will be its own file and there will remain only one plan per run.  The path of the plan file will be in that same directory or be a supplied path as a command line argument.

I’m in the process of reintegrating the environment prechecks for the automated Foster build.

As I do this I can’t help but see how much more readable the testing framework is than the rest of the build controllers.

I’m looking at a rewrite in C for the “controller” and then keeping the executed units in bash subprocesses.

I’m restructuring this all to work in the meantime.

I’ve got a distraction mini-project, also in C.  I’ll be thinking about it.

I’m back and unpacked.

During the move I did some serious thinking about how I want this project to develop and what types of options there were available to get there.

Public Relations Management

I think even addressing the recent political maneuvering around, even as poisonous / malicious as it was and slowed the team down a bit for a week or so, is a mistake– it could be misconstrued as “fighting back” or give the mistaken impression that it is a mutual series of attacks.  Which is exactly the type of intention that was had when these maneuvers were orchestrated.

Too late, already done.

The Linux IRC community, and particularly the Linux IRC community is a haven for all manner of arrogant people — myself included — many of which after a certain level of being established in their communities believe that they are excepted to the obligations of being a good person.

We must make these people small again.  And the way you do that is to build around them, build tall, and show others how to do the same that can benefit the community.

That said, linux.silogroup.org will only contain technical and executive updates to later be used as a basis for tracking decisions and generating documentation.  Public relations efforts, disruptive drama, anything like that, if it even gets brought up, will be moved to my personal blog ( blog.surroindustries.com ) where it should have been.

Potential New Staff Member Recruiting

I am also considering bringing on a dedicated public relations staff member to design, direct, and execute strategies that will benefit the project so that work can be more delegated in this area by specialization.

Infra

The mailing list was where I left off last.  It’s still coming.

This content is password protected. To view it please enter your password below:

Password:

New IRC Arm

Our ad-hoc IRC channel is irc://irc.oftc.net/#surro-linux

Primary discussions will take place on this channel and freenode’s channel will now be used for outreach and support when we eventually make it to release.  This does not preclude the freenode arm from reporting on identified bugs or architectural changes.

It is requested of all greybeards to be in both channels.

I’ll still pop in both channels occasionally but will be using OFTC for my base of operations.

Mailing Lists

We have a candidate to admin the mailing list.  Waiting on confirmation.

 

Well, we’ve run into a snag but we’ve got a plan.

For reasons not yet revealed, after my having been on Freenode for 14 years, it appears a member of Freenode staff has deliberately interfered with the collaborative environment of our IRC channel, unprovoked, where almost all of the architectural discussion has taken place for the Council of Greybeards to Design Surro Linux.

Despite the timeless theme song, this is truly unfortunate and a disservice to the people who have worked so hard on Surro.

I fell into my home office chair tonight after a long day, ready to rerun our build, only to see that our whole domain, surroindustries.com had been klined. My main user and the shell accounts we provide can no longer access Freenode servers unimpeded and we can’t seem to find a member of staff that is able to discuss it.    Sorry about that.

This kind of thing is quite normal on IRC and I see it happen all the time — there is an older generation of linux movers that are absolutely terrified of new projects like this.  Things like this necessitate new projects like ours.

That said, Surro Linux isn’t even slowed down though– not even on IRC.  We’re transitioning to a mailing list model and our channel is still being run.

http://www.surroindustries.com/mailman/listinfo

It is not up and functional yet.  I’ll try to have that up this week.  In the meantime, IRC operations are still up and running.

To be honest, I actually prefer the proposed new setup with a mailing list even better than what we were doing, as the troubleshooting from here will require more careful planning and slower thought to get it right.  This seems to me like it could be more easily accomplished in a mailing list as a medium.

Greybeard DJ_L will be executing the channel with Son_Goku as channel operator.  I feel like I’ve made a good choice and I think they’ll have no problem keeping it alive during my shifting around.

As to motivations of what is, by all reasonable definitions, our attacker– this is yet to be determined as Freenode staff have failed to take responsibility for their decisions.  It is suspected that some social engineering from a widely known local sociopath has taken place for this to even be possible as my standing has been quite good with Freenode since about 2003 and historically they are a very reasonable organization.  Worst case scenario we end up on OFTC.  I dislike the idea of that outcome though, as being on Freenode has been pivotal so far in recruiting and retaining talent for this team.

In short,

1. This changes nothing.  We are in BAU.
2. I’ll be popping in potentially periodically at worst until we are more resilient as an organization against this kind of thing.
3. Builds will resume once we have the mailing lists up which should be about a week to get fine tuned and ready for use.
4. Any information from the community about what is going on with our freenode access would be appreciated: punches.chris@gmail.com

We’re shifting some things around.

 

Hey DJ.

The glibc issue has been worked out.

I needed to introduce a way to:

• Ensure a consistent, reproducible environment between builds.
• Allow multiple-session builds.
• Audit build options and configuration at build script run time.
• Streamline logging for RCA on failures.

And that I did: http://git.luxidolon.com/surro-linux/build-system/blob/fece1e5f911d726a672957ac267ff572e76d72d1/

As you can see the last build is always dated in ISO 8601 format, after which you’ll see the package.  Subpackages are nested though im reconsidering that.

Included in each log set are the make results in build.log, the build_script is the script actually automating that build when the logs were generated, the configure.output.log is the output of the ./configure command where applicable, environment.log is showing the sorted output of env so that the runtime variables can be inspected, make install goes to install.log, and misc.log includes any other actions not already accounted for, mostly health checks on the resultant build targets.

This new addition quickly got surro past the glibc hurdle and on past libstdc++, binutils pass 2, and into gcc pass 2.

This the last failed configure log.  This is happening because I changed the shared libary management scheme mentioned below between the last and current package so the the whole build will need to run:

http://paste.silogroup.org/onefugohet

Luckily because I’m “automating it before doing it” for everything it only takes about an hour to start from square zero now, so we are never really starting from square zero anymore.

I am expecting relatively smooth sailing once we make it out of relinking GCC after this second pass.  I want to get done and out with the compiler tools so I can focus more on package selection.

Determinations were made by the Council Of Greybeards:

For shared libaries and managing architecture coherence we’re going with /lib{,32,64} and merging /usr/* dirs.  This should be simple, straightforward to manage, and hard to break.  /lib will only exist with populated symlinks for a 32bit glibc shared object that is actually a symlink to whats in /lib32.  I am sure this will have slight additions when LSB is introduced, but remember we are building Foster currently which is the system that will build the LSB-compliant Surro-Linux so we actually don’t care about LSB implementation for this round — even if we are going care when we actually boot that environment.

Oddly enough, while I was intending to mimic Red Hat’s directory structure for shared library management and avoid old Debian/Gentoo/Arch Linux FS trees , this is actually the functional equivalent of being right in between both models as Red Hat uses /lib{,64} and older Debian and Arch uses /lib{,32} each for opposing architectures, and neither of which feels sane to me except as legacy footprints– although I am perfectly willing to admit their maintainers know much more about this topic than I do.

Regarding multilib, I’ve got some reading to do on this topic to be more informed.  Here’s what’s on my plate:

1. https://wiki.debian.org/Multiarch
2. https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
3. https://www.freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/

Luckily for us, this change will be both FHS and LSB compatible so we will be able to carry the same design into SURRO if it works out well.

I just wanted to post updates now that we’ve had some as I’ve been caught up with managing life things and haven’t had much time to work on this.  Once foster is bootable and the automations are complete I’ll be opening up the VCS to make this a collaborative effort.