code-augmentor-core

Provides common functionality to related plugins and command line tools in bringing code generation facilities to every programmer

License	License MIT License
GroupId	GroupId com.aaronicsubstances
ArtifactId	ArtifactId code-augmentor-core
Last Version	Last Version 2.1.0
Release Date	Release Date Aug 22, 2020
Type	Type jar
Description	Description code-augmentor-core Provides common functionality to related plugins and command line tools in bringing code generation facilities to every programmer
Project URL	Project URL https://github.com/aaronicsubstances/code-augmentor
Source Code Management	Source Code Management https://github.com/aaronicsubstances/code-augmentor

Download code-augmentor-core

Filename	Size
code-augmentor-core-2.1.0.pom
code-augmentor-core-2.1.0.jar	105 KB
code-augmentor-core-2.1.0-sources.jar	67 KB
code-augmentor-core-2.1.0-javadoc.jar	175 KB
Browse

How to add to project

Apache Maven

<!-- https://jarcasting.com/artifacts/com.aaronicsubstances/code-augmentor-core/ -->
<dependency>
    <groupId>com.aaronicsubstances</groupId>
    <artifactId>code-augmentor-core</artifactId>
    <version>2.1.0</version>
</dependency>

Gradle Groovy

// https://jarcasting.com/artifacts/com.aaronicsubstances/code-augmentor-core/
implementation 'com.aaronicsubstances:code-augmentor-core:2.1.0'

Gradle Kotlin

// https://jarcasting.com/artifacts/com.aaronicsubstances/code-augmentor-core/
implementation ("com.aaronicsubstances:code-augmentor-core:2.1.0")

Apache Buildr

'com.aaronicsubstances:code-augmentor-core:jar:2.1.0'

Apache Ivy

<dependency org="com.aaronicsubstances" name="code-augmentor-core" rev="2.1.0">
  <artifact name="code-augmentor-core" type="jar" />
</dependency>

Groovy Grape

@Grapes(
@Grab(group='com.aaronicsubstances', module='code-augmentor-core', version='2.1.0')
)

Scala SBT

libraryDependencies += "com.aaronicsubstances" % "code-augmentor-core" % "2.1.0"

Leiningen

[com.aaronicsubstances/code-augmentor-core "2.1.0"]

Dependencies

runtime (1)

Group / Artifact	Type	Version
com.google.code.gson : gson	jar	2.8.6

Project Modules

There are no modules declared in this project.

Code Augmentor

Description

The CodeAugmentor tool generates code for a programmer in such a way that it appears to the rest of the build process that the programmer typed it himself. This enables tactics in a programmer's toolbox that can be leveraged to maintain an emergent, fragile or aging software architecture and still deliver expected quality guarantees to stakeholders.

To accomplish this, data-driven programming mindset is neccesary. From Eric Raymond's The Art of Unix Programming book, this alternative definition of the term "data-driven programming" is intended:

When doing data-driven programming, one clearly distinguishes code from the data structures on which it acts, and designs both so that one can make changes to the logic of the program by editing not the code but the data structure.

The tool heavily relies on JSON as the standard data exchange format of choice.

Operationally, the tool has two aspects: one part which resembles a preprocessor (such as the C/C++ preprocessor), and another part which resembles a linter (such as ESLint). However there are important differences:

Like a C/C++ preprocessor, one part of CodeAugmentor is about code generation. In CodeAugmentor however, code generation is the main goal. It shares that same goal with C/C++ preproccessor macro exapansion facility, and thus with any general purpose macro processor (e.g m4). It is not about header file inclusion and conditional compilation which are offered by the C/C++ preprocessor.
CodeAugmentor enables the programmer to employ JSON and a scripting language to generate code, removing the need to learn a dedicated macro processor language. In principle any scripting language can be employed with some small setup implementation required. However the following languages have been given priority by having their setup implementations implemented: Groovy and other JVM languages, NodeJS, Python 3 and PHP 7.
Unlike a C/C++ preprocessor which modifies code "under the hood", CodeAugmentor modifies the programmer's source code "in place" from external code generator scripts, and synchronizes the generated code with the scripts. Any changes to the scripts, or any tampering of generated code by programmer will trigger an out-of-sync error.
Like a linter, the other part of CodeAugmentor validates code without modifying it. This enables straightforward integration of this part with build tools, as hooking a custom step into most build tools is far easier if the step doesn't modify any source code.

The workflow of the programmer using this tool may be exemplified in the following manner:

Code to be generated into source files is written into single-line comments as JSON.
Programmer writes code generator scripts in a related directory, and hooks it up to CodeAugmentor.
CodeAugmentor is integrated into build process as the last "manual step" (e.g. before compilation, before transpilation, or at runtime initialization) to check whether source files need to be regenerated.
During build, CodeAugmentor may report that certain source files are out of sync and need regeneration, and cause build to fail. It would have generated the newer source files into a temporary location.
Overwriting out-of-date source files with newer ones from temporary location is the programmer's responsibility, through the use of shell/batch scripts provided by CodeAugmentor.
Build process is triggered again, and this time CodeAugmentor doesn't complain, confirming that source files are now in sync with code generated by programmer's scripts.

Use Cases

See Background wiki page. There you will find the motivation for developing this tool and its intended use case scenarios.

Published Artifacts

Standalone command line application released here for use with any programming language, with the requirement of Java 8 JDK or JRE.
Ant, Maven and Gradle plugins available on Maven Central and Gradle plugin portal, for scripting with Groovy and for easier integration with Java and Android IDEs.
Packages for officially supported languages available for NodeJS, Python 3 and PHP 7.
Javadoc available online for code-augmentor-core.

How It Works

The operation of Code Augmentor is achieved in three stages: preparation, processing, and completion.

During preparation, the source files passed to Code Augmentor are tokenized into lines, and then lines with special prefixes known as directives are identified, and classified into augmenting code sections, generated code sections or code sections to be skipped over by Code Augmentor.

An augmenting code section generally consists of 3 parts: JSON data, raw string data, or code string. Code strings are meant to be evaluated as code in some scripting language (such as Groovy, NodeJS, Python or PHP). The expected and recommended structure of an augmenting code section is that it consists of an initial code string line which is the name of a procedure in a scripting language, and then optionally following it are either JSON or raw string data which are the arguments to be passed to the procedure.

The preparation stage tokenizes source files and retrieves the augmenting code sections in them, saving them into two kinds of files: prep file and aug code file. prep file is a file full of numbers needed by completion stage, and is not of interest to the script writer, except for script writer to pass it around to whatever stage requiring it.

aug code file is passed to processing stage. Generally for each augmenting code section, code generator scripts have to produce content for its generated code section. The output of this stage is referred to as gen code file.

Completion stage uses prep file and gen code file to insert generated code content into generated code sections for each augmenting code section. During this stage generated code sections are created if none existed previously where insertions must be made.

If code change detection remains enabled, then completion stage identifies source files whose content is not equivalent to that obtained after inserting generated code content. This could be because there was no generated code content at all, or the generated code content which used to be there is different from the incoming one. Plugins and command line application will normally turn any code change detected into an error for programmer to reconcile using shell scripts provided by Code Augmentor.

Further information helpful to script writers is available on related wiki page.

Getting Started with standalone command line application

Ensure you have Java 8 JDK or JRE installed on your computer, and that java command is on the system path. Run java -version from command line to verify, and should get output resembling the following:
```
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
```
Download the latest release of code-augmentor-app, decompress/unarchive download, and place the bin folder containing the code-augmentor-app script in your system path. Alternatively you can always use the full path to call code-augmentor-app script directly. In the next steps it is assumed code-augmentor-app is in your system path.

Verify ability to get help text of script by running code-augmentor-app -h, and should get an output resembling the following:

usage: code-augmentor-app -f <ant build file> [-h] [-t <ant target>] [-v]
 -f,--file <ant build file>   use given Ant build XML file
 -h,--help                    print help information and exit
 -t,--target <ant target>     use given Ant build target
 -v,--version                 print version and exit

Decide on which scripting language you are going to employ for writing code generator scripts: NodeJS, PHP 7, Python 3, Groovy. Strictly speaking more than one language can be used, but the expected usual case is usage of one scripting language.
Sample projects are available for getting started: NodeJS examples, PHP 7 examples (requires Composer), Python 3 examples (requires Poetry), Groovy examples. Download files in the example directory for the chosen scripting platform.
Each sample project has a build.xml which runs Code Augmentor on files of a src subdirectory beside build.xml. Assuming current directory is the directory containing the build.xml file, first run
```
code-augmentor-app -f build.xml -t check-exec
```
to ensure scripting platform is setup properly (ie have run npm install, composer install, poetry install, etc).

Note:

a. For code synchronization examples, a tempSrc directory is used instead of src directory so changes can be made to source files without touching original source files.

b. tempSrc is deleted and recreated each time check-exec Ant target is executed, so be sure to call it just once per a series of code synchronization experiments.

c. check-exec Ant target and tempSrc directory only exist for the purpose of this getting started tutorial, and are therefore not needed in real world usage of Code Augmentor.
Now launch full Code Augmentor operations by running
```
code-augmentor-app -f build.xml
```
By default Code Augmentor is configured to detect code changes in generated code sections. If code_aug_complete task in build.xml is configured to disable code change detection, like this:
```
<code_aug_complete codeChangeDetectionDisabled="true" />
```
then Code Augmentor operation ends, with files in generated subdirectory (beside build.xml) corresponding to files in src subdirectory.
Else Code Augmentor will complain/fail with (normal) error that files are now out of sync. Investigate generated subdirectory beside build.xml, and some text files and shell scripts should be present.

a. CHANGE-SUMMARY.txt - contains paths in tempSrc folder with out-of-sync changes.

b. CHANGE-DETAILS.txt - contains information on what actually changed in Unix diff normal format.

c. EFFECT-CHANGES - shell script to run to update files in tempSrc with generated files.

d. A directory will be present as well for each srcDir FileSet specified in build.xml
Run EFFECT-CHANGES -f and observe that files in tempSrc subdirectory have been overwritten with corresponding files from generated subdirectory.
Now rerun
```
code-augmentor-app -f build.xml
```
and Code Augmentor should now end its operation successfully without any generated files, indicating that code generator scripts and files in tempSrc subdirectory are now in sync.
One can at this stage experiment with the synchronization mechanism by editing augmenting code sections inside tempSrc of example project directories with -sync suffix, running code-augmentor-app to verify detection of changes, running EFFECT-CHANGES, and rerunning code-augmentor-app to verify synchronization.

Automated Build Hook Points

In practice it is desired to automate synchronization mechanism by setting up code-augmentor-app -f build.xml to be called just before the following common build steps, whichever is the earliest of compilation, transpilation or runtime initialization.

a. For Apache Maven, the validate phase is perfectly suitable for attaching code-augmentor:run goal.

b. For Gradle, letting compileJava task depend on codeAugmentorComplete or codeAugmentorRun task will do.

c. In general clients of Code Augmentor have to discover for themselves suitable build hook points for automating synchronization mechanism.

Some guidance however is in order: Some build tools provide a pre-build hook point which may be all that is needed. Microsoft Visual Studio IDE for example provides pre-build events for such purposes.

One practical challenge is that some build tools do not invoke their pre-build stage if compiled/transpiled code is up to date with source code files (e.g. Microsoft Visual Studio). This is undesirable for CodeAugmentor, since code generator scripts can invalidate source code files at any time, with build tools completely unaware.

I suggest a generic solution to this problem by leveraging post-build hooks (in addition to the mandatory pre-build hooks) and a source file per project dedicated specifically for this purpose. Assuming that code generator scripts are going to be managed in a version control system like Git, then dedicated source code file can record the latest full commit identifier. Any time the code generator scripts change, a new commit can be made, making dedicated source code file contents out of date. Thus all that is needed is a post-build hook which can get the current commit identifier of the code generator scripts and compare it with the dedicated source code file contents.

This repository provides working post-build hook scripts for Windows and Linux platforms in the examples directory. The current scripts propose the following:

Set up post build hook to pass dedicated source code file path as first command line argument. Also set working directory of script launch to directory of code generator scripts.
Supply a command to run to get latest version information of the code generator scripts, such as commit identifier of the code generator scripts folder under version control.
Supply lines of code which must surround the output from the configured command in order to make the dedicated source file contents valid, before and after the command output. Empty lines are currently not allowed due to Windows batch scripting limitations.

With these in place it is expected that automating CodeAugmentor synchronization checks will be straightforward. The scripts are meant to be copied into individual projects and adapted accordingly for use in a wide range of scenarios.

Documentation of Plugins

Development Environment

Java 8 (Oracle Java SE 1.8.0_152, AdoptOpenJDK 1.8.0_265, Zulu 1.8.0_252)
Apache Ant 1.10.1
Apache Maven 3.5.2
Gradle 5.6.4
Windows 10 Professional
CentOS Linux 8
Visual Studio Code 1.42.0
Groovy 2.4.13
NodeJS v12.13.0
Python 3.8.1
PHP 7.4.5 (cli)

Build Instructions

Ensure the following are installed locally: JDK 8, Apache Maven, Groovy. Ensure JAVA_HOME and GROOVY_HOME environment variables are set up properly.
Also ensure javac, java, mvn and groovy executables are placed on the system path.
Clone repository and run gradlew alltests from root of project. Can also run gradlew clean alltests instead (Gradle 5.6.4 will be downloaded and used by the commands). This tests and builds all the Java projects, and can take up to 6 minutes to complete.

Tests Requiring Manual Inspection

The following tests (currently all in the code-augmentor-core project) generate standard output/error messages which can be verified for correctness:

core.tasks.*
core.util.CodeGenerationRequestCreatorTest.{testValidateAugCodeSection,testIdentifyAugCodeSectionsForErrors}

Tests involving non-Java supported languages

In order to run further tests involving supported non-Java languages in the process-aug-code-tests directory, ensure the following are installed: NodeJS, Python 3, PHP 7 (CLI).

Ensure node, python/python3 and php executables are placed on the system path.
Clone the repositories for each of the supported languages from their respective repositories.
In the process-aug-code-tests directory, make a copy of LocalConfig.groovy.sample and name it LocalConfig.groovy. Edit the properties declared in it to match the locations of your supporting package project directories. Paths relative to Java repository can be used.
(Optional) Run gradlew :code-augmentor-core:build from root of Java project. This can be skipped if Java projects have been built recently.
Make process-aug-code-tests the current directory from command line and use groovyw tests.groovy to run tests involving non-Java packages. If all tests pass, should see 0 failures.

Versions

Version
2.1.0 Aug 22, 2020
2.0.0 Jun 20, 2020
1.0 May 27, 2020

code-augmentor-core

License

GroupId

ArtifactId

Last Version

Release Date

Type

Description

Project URL

Source Code Management

Download code-augmentor-core

How to add to project

Dependencies

runtime (1)

Project Modules

Code Augmentor

Description

Use Cases

Published Artifacts

How It Works

Getting Started with standalone command line application

Automated Build Hook Points

Documentation of Plugins

Development Environment

Build Instructions

Tests Requiring Manual Inspection

Tests involving non-Java supported languages

Versions