Kevin Day [Mon, 8 Aug 2022 04:04:26 +0000 (23:04 -0500)]
Update: The utf8 program should be using the stream read functions.
The file is opened using the stream open functions.
It is silly and inconsistent to use the non stream functions to read from a file opened via a stream.
Kevin Day [Mon, 8 Aug 2022 04:00:46 +0000 (23:00 -0500)]
Bugfix: Functions private_f_file_flush() and private_f_print_safely_get() are not being inclided when needed.
The disable macro _di_f_file_stream_close_ is missing.
The disable macros _di_f_print_raw_safely_, _di_f_print_raw_safely_dynamic_, and _di_f_print_raw_safely_dynamic_partial_ are missing.
Kevin Day [Sun, 7 Aug 2022 01:53:18 +0000 (20:53 -0500)]
Bugfix: Stand alone build is pulling headers from system.
The projects headers should be pulled rather than system headers.
Restructure the stand alone package structure to match the installed headers structure.
This allows for leveraging the pre-compilers ability to locate the files.
Pass '-I sources/c' to the flags to properly locate the local headers rather than the system headers.
Kevin Day [Sun, 7 Aug 2022 01:02:20 +0000 (20:02 -0500)]
Update: The test script not uses stand alone mode for compiling Featureless Make.
The Featureless Make program that is compiled and used to run the tests is now done so using the stand alone mode.
This simplifies the process and avoids having any libraries in the path that could potentially be used (this has never happened, by the way).
Kevin Day [Sun, 7 Aug 2022 00:28:48 +0000 (19:28 -0500)]
Update: Implement stand alone build for Featureless Make.
This functions as both an example as well as a functional program.
I have not yet automated the disable macro process.
This process has been manually performed and may not remove all of the unnecessary code.
This does remove a vast majority of the unnecessary code.
Kevin Day [Sun, 7 Aug 2022 00:24:32 +0000 (19:24 -0500)]
Feature: Add support for 'stand_alone' builds in the package script.
Provide a way to compile programs, such as Featureless Make, with all of their dependencies and most (if not all) of the unnecessary dependencies not part of the programs.
These programs, such as the Featureless Make program, are compiled as a single program with no libfll.so (or program library such as libfake.so) built.
These are still compiled as either shared or static.
All non-FLL libraries in the shared programs are still linked as libraries.
This is a feature in the helper script and does not result in any API (or ABI) breakage.
Kevin Day [Sun, 7 Aug 2022 00:14:01 +0000 (19:14 -0500)]
Bugfix: Mistakes in disable macros and remove redundant check.
Fix numerous disable macro mistakes.
There are several functions that do not have the correct amount of disable macros.
There are several disable macros that have mistakes in the name structure.
The functions private_f_utf_string_append() and private_f_utf_string_append_nulless() are accidentally defined in the dynamic.h header file in addition to the string.h header file.
These should only be defined in one location and the ones defined in dynamic.h are incorrectly located.
Remove redundant errno check for ENOMEM in private_fl_directory_list().
Kevin Day [Fri, 5 Aug 2022 02:46:03 +0000 (21:46 -0500)]
Update: Change logic to avoid potential negative numbers.
Avoid using a subtraction.
Instead just use a comparison.
This should be more performant and also avoids potential negative numbers when dealing with unsigned numbers.
Kevin Day [Tue, 2 Aug 2022 22:53:13 +0000 (17:53 -0500)]
Update: Redesign behavior of pipe, allowing it to run properly inside software like Gitlabs CI or Githubs CI.
I believe both of the CI systems (Github and Gitlab) use Docker.
These systems will place a new line in the input pipe before starting the programs being called.
This is bad behavior but I cannot do anything about it.
The previous design of Featureless Make is to use only the input pipe if it is specified.
This results in the Featureless Make operations to fail.
This redesigns the pipe handling behavior of Featurleess Make rather than try to submit a bug report to a team that is likely to completely ignore me.
The new behavior is as follows:
- If a pipe is specified, treat the pipe is prepended to the buffer that will be used for processing the file.
- This works for both "build" operations and "make" operations.
- The input pipe is treated as a "fakefile" when using the "make" operation.
- The input pipe is treated as a "settings" file when using the "build" operation.
- The input pipe may have an error bit so always clear the error.
- The "build" and "make" operations can be called within the "make" operation, recursively, so the pipe must only be processed by the outermost fakefile.
- The operation needs to be detected and identified as "default" to better determine how to handle missing files when a pipe is present and when a pipe is not present.
- Avoid resetting the buffer on every load so that the input pipe can be prepended without being reset.
Some of the code is cleaned up and simplified in regards to files.
Only perform the pre-process cleanup checks when there are more than two operations.
Kevin Day [Mon, 1 Aug 2022 03:54:48 +0000 (22:54 -0500)]
Update: Enable gitlab test support.
The gitlab can be designed to work the same as github.
Change the name of the relevant code from "github" to "ci" and have both gitlab and github modes call the ci functions.
For the time being using github as the source code repository is fine.
I should eventually clone the respective repos in gitlab and then conditionally use the URLs based off of the given build mode (github or gitlab).
Kevin Day [Sun, 31 Jul 2022 22:50:30 +0000 (17:50 -0500)]
Bugfix: Block is incorrectly being reset when an else condition precedes an if condition.
There is a logic mistake where the "else" is not being considered when determining when to reset the block.
This results in the block states being reset when the previous operation is an "else" condition.
This results in the second else condition potentially running even if the prior condition already ran.
This was discovered when investigating Github actions test failures that pass locally.
The Github actions tests use a special test argument that I had not tested locally with.
Kevin Day [Sat, 30 Jul 2022 00:22:10 +0000 (19:22 -0500)]
Bugfix: private_f_file_open_at() has an incomplete set if ifdef conditions.
The ifdef conditions for disabled functions is incomplete.
This results in the private_f_file_open_at() becoming incorrectly unavailable when the appropriate disable defines are set.
Kevin Day [Thu, 28 Jul 2022 00:36:31 +0000 (19:36 -0500)]
Update: Improve specification files.
Expand the vocabulary to include more words such as "character".
This allows for more granular control over substitution.
This is a must to easily convert to both plain text and HTML from the same source file.
Kevin Day [Thu, 28 Jul 2022 00:09:16 +0000 (19:09 -0500)]
Update: Clarify IKI standard in regards to delimits and improve specification file.
Add additional commands to the specification iki_read example.
Expand the vocabulary to include more words such as "character".
This allows for more granular control over substitution.
This is a must to easily convert to both plain text and HTML from the same source file.
Fix incorrect example where the delimits are on the wrong characters (url:\' is not correct and instead should be url\;').
Add delimits to accidentally undelimited code.
These changes exposed a mistake in both the processing code and a detail on how I explained the specification.
The IKI processing only needs a delimit when the vocabulary name followed by a colon followed by either a single or a double quote.
That is to say, the matching closing single or double quote is irrelevant.
This simplifies the logic needed to both read and write the IKI content for both machines and humans.
Note that this is in contrast to how the FSS works.
IKI is designed for different purposes than FSS and thus has slightly different design paradigms.
The code processing this is incorrectly failing to apply delimits for any potentially valid IKI content inside an already valid but delimited IKI variable.
Fixing this conveniently also results in simpler code and fewer variables used (saving trivial memory).
Kevin Day [Sun, 24 Jul 2022 03:26:25 +0000 (22:26 -0500)]
Update: Add +E/++error parameter to firewall.
The previous security commit 7348813d0af17acda440eb66c038f6bdd07c1a6e is the result of an incomplete addition of the +E/++error parameter.
The size of the total parameters was incremented to ensure space for the +E/++error but the +E/++error was never added.
This increments the total parameters and adds the +E/++error row.
Kevin Day [Sat, 23 Jul 2022 22:41:30 +0000 (17:41 -0500)]
Bugfix: Link operation is incorrect, support new link arguments, and clone and copy operation failure problems.
The link operation target path is relative to the point path.
The previous code is trying to treat the target path in isolation.
Change the behavior to make the target path relative to the point path.
Add two new options to make creating symbolic linking easier to use:
1) "force": Used to forcibly overwrite an existing file or directory.
2) "strict": Used to require the target path to exist when creating the symbolic links.
The clone and copy operations now have better error return code processing.
The operate process type operations are not all in their own functions.
Change the code to move all of the operate process type operations into their own functions.
Kevin Day [Sat, 23 Jul 2022 15:55:55 +0000 (10:55 -0500)]
Update: Organize the validate operation into separate type functions and improve error handling logic.
Move the contents of each validate operation if condition block into separate functions.
This makes the validate operation code more consistent with how the process operation code.
This simplifies some of the nesting.
This has a cost of adding and calling more functions.
Multi-purpose functions are used where applicable rather than having a single function for each operation type.
The error handling is improved in some cases where errors are printed for all arguments rather than the first.
This is done only in cases where I could obviously and quickly determine that this can be done.
There is the possibility of mistakes or regressions due to the structural changes and the condition block changes.
The commit e70631a5463f7dcb7ad70a0f92040c2d59ad947a also went a bit over kill and added unecessary trailing slashes.
I only tested directoreis at the time and it was not a problem.
This broke for non-directory file types.
Kevin Day [Thu, 21 Jul 2022 22:44:14 +0000 (17:44 -0500)]
Update: Timeout changes and documentation updates.
I seem to have forgotten to complete this functionality.
I believe it is too much work to do and this will not make the stable release.
Update the documentation to reflect this.
Add a new timeout option called "exit".
This allows for handling the timeout when exiting the program.
This is implemented and in use.
The implementation is dirt simple and sub-optimal but allows for the functionality to be quickly implemented.
The important addition here is the ability to disable the respective timeouts.
No child processes are killed when the exit timeout is disabled.
This can result in the program stalling for a long time or indefinitely.
A new flag property is added to controller_entry_t and controller_entry_action_t.
Kevin Day [Wed, 20 Jul 2022 03:19:51 +0000 (22:19 -0500)]
Feature: Add missing signal functions f_signal_action(), f_signal_pause(), and f_signal_suspend().
The f_signal_pause() just calls pause() and returns F_none.
The return value of pause() is meaningless if I am understanding the man pages correctly.
It should always return -1 and then sets errno despite this not being an actual error.
Kevin Day [Wed, 20 Jul 2022 01:47:01 +0000 (20:47 -0500)]
Cleanup: Remove bad path part from string in testfiles.
The 'parameter:' is not valid and does nothing here.
This is likely the result of a copy and paste that didn't fully delete the parameter IKI variable.
Kevin Day [Tue, 19 Jul 2022 03:35:29 +0000 (22:35 -0500)]
Bugfix: Guarantee that the child processes are terminated.
The terminate signal is not being received for unknown reasons.
The signals are unblocked for the child process.
And yet, it continues to block.
The final part of the terminate due to signal is to forcibly terminate (sending the kill signal).
This is now happening for child processes.
Sending the kill signal allieviates the problem where the controller program will not terminate.
This fix causes the reduces the severity of the problem such that the terminate should still terminate.
However, this fix does not solve the problem with the terminate signal not being received by a child process that is not blocking the signal as far as I can tell.
Kevin Day [Mon, 18 Jul 2022 03:36:55 +0000 (22:36 -0500)]
Bugfix: The copy operation doesn't work properly with trailing slashes.
The generated path might not be valid when a trailing slash is supplied on the source or the destination.
Increase the static array size to accommodate at least 2 added separators.
Only add the separators if one does not already exist.
This does not attempt to clean up the code if there are multiple separators beyond the first.
(That is to say "copy a/// b///" will not become "copy a/ b/".)
Kevin Day [Mon, 18 Jul 2022 01:26:23 +0000 (20:26 -0500)]
Bugfix: IKI substitution is incorrectly applied.
The function controller_rule_action_read_delimit_apply() is remapping the array and applies the delimits.
Another function is then applying the same delimits again, but against the remapped buffer.
This results in the delimits being applied to the wrong locations in the string.
Get rid of the controller_rule_action_read_delimit_apply().
The use of this function results in an incorrect string when printing validation.
Thoroughly clear the entire IKI data when processing the actions.
Kevin Day [Sun, 17 Jul 2022 23:42:21 +0000 (18:42 -0500)]
Security: Invalid reads on NULL terminated strings with a max width greater than 1.
The max width cannot be reliably trusted in NULL terminated strings.
Check if each character in the expected max width is not terminated early due to a NULL.
If it is, then adjust the max width.
Kevin Day [Sun, 17 Jul 2022 20:49:48 +0000 (15:49 -0500)]
Update: Reduce number of allocations by pre-determining size when easily possible.
There are several cases where the expected size can be calculated.
Pre-allocated the expected size to reduce reallocation and improve performance by reducing memory overhead.
I also noticed that the comment is no longer correct and that the not quoted check could be reversed.
Changing this simplified the code, slightly.
Kevin Day [Sun, 17 Jul 2022 20:34:16 +0000 (15:34 -0500)]
Bugfix: Reserved parameter IKI expansion is not adding spaces and the "top" reserved parameter should always end in a slash.
The IKI expansion on reserved parameters, such as "parameter:"fakefile"", should include spaces as appropriate.
For example, given a call to "fake -f my_fakefile", the parameter:"fakefile" should expand into "-f my_fakefile" but is instead expanding into "-fmy_fakefile".
For security reasons, the expanded paths, such as parameter:"top", should always have a trailing slash.
Consider "rm -Rf parameter:"top"tmp" vs "rm -Rf parameter:"top"/tmp".
On the left side, if parameter:"top" resolves into an empty string, then the command is: "rm -Rf tmp".
On the right side, if parameter:"top" resolves into an empty string, then the command is: "rm -Rf /tmp".
The right side would end up destroying a path outside of the project root, such as "/tmp"!
Kevin Day [Sun, 17 Jul 2022 03:09:09 +0000 (22:09 -0500)]
Update: Add support for +E/++error, change '+q' to '+Q', and fix some past tense words.
Selecting a quiet mode that still prints errors is very helpful in embedded fakefiles inside of controller rules.
I noticed that almost all of the verbosity related parameters are upper case except for the "quiet" parameter.
Change the "quiet" parameter from "+q" to "+Q".
I noticed some words in the past tense.
The goal is to use present tense.
Using past tense a habit I hope to get out of when programming.
There parameters are not always expanding properly resulting in the parameter being empty, having the incorrect order, or being improperly merged with another parameter.
Change the array resize behavior to resize after incrementing the arguments rather than before.
Detect and handle special cases where separation needs to be applied and when separation does not need to be applied.
Remove random space that is being accidentally appended when printing arguments (probably an accident from a previous commit).
settings:
parameter a iki <-assure_space unassure_space->
parameter b value
main:
if exist 'parameter:"a"'
print yes (parameter:"a")
else
print no (parameter:"a")
print 0 parameter:"a"
print 1 parameter:"b"
print 2 parameter:"b".
print 3 "parameter:"b""
print 4 "parameter:"b\" between parameter:"b""
print 5 'begin parameter:"a" middle parameter:"a" end'
print 6 "begin parameter:"a\" middle parameter:"a\" end"
print 7 begin parameter:"a" middle parameter:"a" end
print 8 begin parameter:"a"! middle parameter:"a"@parameter:"a" end
Should produce results like:
no (iki <-assure_space unassure_space->)
0 iki <-assure_space unassure_space->
1 value
2 value.
3 value
4 value between value
5 begin iki <-assure_space unassure_space-> middle iki <-assure_space unassure_space-> end
6 begin iki <-assure_space unassure_space-> middle iki <-assure_space unassure_space-> end
7 begin iki <-assure_space unassure_space-> middle iki <-assure_space unassure_space-> end
8 begin iki <-assure_space unassure_space->! middle iki <-assure_space unassure_space->@iki <-assure_space unassure_space-> end
Move the relevant arguments and iki data into a shared cache to save memory consumption.
Rename path_cache to cache_path for consistency.
Kevin Day [Fri, 15 Jul 2022 04:27:43 +0000 (23:27 -0500)]
Bugfix: Single quotes are not being properly detected in FSS Extended Read functions.
A copy and paste mistake where f_fss_quote_type_double_e when instead f_fss_quote_type_single_e should be used resulted in the quote being set to NULL.
Kevin Day [Thu, 14 Jul 2022 02:05:08 +0000 (21:05 -0500)]
Update: Change "exists" to "exist" in fakefile syntax.
The use of "exists" is grammatically correct and the use of "exist" is grammatically incorrect.
This is not in English grammar.
The practices of this project are to focus on using "s" strictly for plural.
The practices of this project are to use simple or base words more often.
The area in which proper grammar is allowed is when interacting with the user rather than with code.
A project like Fake has a target user who is a programmer.
This is a grey area.
The project is already using "if define" rather than "if defined".
For the purpose of keeping a consistent design, I am favoring "exist" over "exists" for this grey area.
Another bonus is that "exist" is shorter than "exists" (however trivial).
Kevin Day [Wed, 13 Jul 2022 23:19:21 +0000 (18:19 -0500)]
Update: Strip out NULL characters after applying delimits.
Once a rule is read and the IKI data is parsed, apply the IKI delimits.
NULL characters replace the delimits.
Strip out all NULL characters from the string after the delimits are applied.
Kevin Day [Wed, 13 Jul 2022 12:05:32 +0000 (07:05 -0500)]
Bugfix: Delimits are not being applied for IKI variables.
The controller program is not applying the delimits for would-be-valid IKI variables.
This becomes a huge problem when these would-be-valid IKI variables are passed to the fake program.
If the would-be-valid IKI variables are properly delimited, then the fake program would see them as valid IKI variables.
Kevin Day [Wed, 13 Jul 2022 12:01:39 +0000 (07:01 -0500)]
Bugfix: IKI variables are incorrectly being processed when there is a non-IKI IKI-like string.
The delimits, after the first, are not being processed.
This is the result of an accidental double increment.
When checking against a possible IKI variable and it is determined that the string cannot be an IKI variable, a double increment occurs.
What is happening is that the break statement only breaks out of the immediate loop.
There is a second loop that does an increment is not being broken out of.
Utilize the separator_found boolean to determine whether or not to perform the additional break.
Kevin Day [Mon, 11 Jul 2022 02:45:33 +0000 (21:45 -0500)]
Security: Invalid read for formatted printing using partial ranges on a string.
If the start position is greater than the used buffer, then an invalid read occurs.
Properly verify that the start position is not greater than or equal to the used length of the string.
Kevin Day [Sun, 10 Jul 2022 22:26:42 +0000 (17:26 -0500)]
Cleanup: Converted type is actually uint32_t rather than f_utf_char_t.
The f_utf_char_t is supposed to be an uint32_t so this is not a problem.
The intent and design of this, however, is that f_utf_char_t is a special case representing the character as a string rather than as a digit.
The f_utf_char_t is stored as a 4-byte integer to store each byte representing a character.
The uint32_t is simply a straight up 4-byte integer.
This is the numeric value of the code point rather than the representation as a string.
This is an important semantic difference.
Kevin Day [Sun, 10 Jul 2022 22:10:52 +0000 (17:10 -0500)]
Feature: The featureless make program now supports the "write" operation.
This is an oversight on my part.
There should be an operation to write to a file.
There are two forms of this new "write" operation.
1) Truncate a file (deletes all data within a file).
2) Append to a file.
A file is created if it does not already exist in both cases.
The "write" operation supports some standard escape sequences as well as some non-standard ones.
Standard Escape Sequences:
- "\f": Form Feed.
- "\n": New Line.
- "\r": Carriage Return.
- "\t": Tab.
- "\v": Vertical Tab.
- "\\": Backslash Character (may require additional slashes in certain circumstances.)
- "\0": NULL Character.
Non-Standard Escape Sequences:
- "\U+": Unicode Sequence (followed by a valid Unicode sequence with a minimum 4 hexidecimal digits and a maximum of 6 hexidecimal digits).
- "\U-": Terminate a Unicode Sequence, allowing for "\U+000A\U-5" to be equivalent to "\n5".
Kevin Day [Sun, 10 Jul 2022 05:45:05 +0000 (00:45 -0500)]
Bugfix: Incorrect information is printed on certain errors.
Remove unused line variable.
The f_fss_count_lines() function appends to the calculated length variable.
The number is not being reset.
This results in each iteration adding to the previous:
Line number 1, count = 1.
Line number 2, count = 3.
Line number 3, count = 6.
etc...
Reset the line number on each pass of the loop to get the correct line number.
An error message is printing "1" when it should be printing "2".
The "%Q" should be used instead of "%s" for the static string.
Replace "parameter" with "Content" to be consistent with other error messages.
Kevin Day [Sun, 10 Jul 2022 02:52:07 +0000 (21:52 -0500)]
Refactor: "if defined" and "if not defined" to be easier to use in fake program.
Using "if not defined parameter work" can be confusing.
Using "if not parameter work" is shorter and easier to understand.
Using "if defined environment PATH" can be very confusing.
Using "if define PATH" is shorter and a lot easier to understand.
Break apart the "if defined" (and "if not defined") logic into two operations:
1) "if define" (and "if not define").
2) "if parameter" (and "if not parameter").
This makes the behavior easier to understand as it directly maps to the "defined" and "parameter" settings.
Kevin Day [Sat, 9 Jul 2022 22:59:59 +0000 (17:59 -0500)]
Update: Use "settings" instead of "setting" for better consistency between fake program and controller program.
Featureless Make is using "settings" and the Controller program is using "setting".
Fix this inconsistency.
The term "settings" sounds more accurate than "setting".
Kevin Day [Sat, 9 Jul 2022 22:44:24 +0000 (17:44 -0500)]
Update: Restrict environment to PATH and LD_LIBRARY_PATH by default.
Change all of the setting files and fakefiles to restrict the environment variables.
Only PATH and LD_LIBRARY_PATH are exposed so that custom build environments can easily be used by default.
Kevin Day [Sat, 9 Jul 2022 22:02:15 +0000 (17:02 -0500)]
Update: Make environment variable handling design consistent between fake and controller programs.
The Featureless Make system does not have a way of passing all environment variables.
Add a flag to designate whether or not environments is empty because it is not defined or is empty because it is defined as empty just like the controller program does.
This then allows for more flexible control over whether or not the environment variable security.
Update the example setting files and fakefiles to expose PATH and LD_LIBRARY_PATH by default.