Kevin Day [Thu, 9 Jul 2020 05:13:15 +0000 (00:13 -0500)]
Progress: add support for determining quotes in use, other fixes and cleanups.
While not enthusiastic about this, I believe that I need to report the quote used to the caller.
Make this an optional parameter.
This took some significant consideration.
Not all standards utilize quotes and I also do not want to make FSS object and content types more complex than ranges.
Using a separate variable, while a bit more tedious in having to maintain consistency, is an acceptable approach.
FSS-0000 objects, FSS-0001 objects, and FSS-0001 contents all use the same logic for identifying their types.
Reduce code by using the same function.
Because FSS-0001 contents is an array of contents, move delimit processing outside of the private function.
Replace (used + 1 > size) checks with (used == size) checks to increase performance by avoiding arithmetic operations.
Replace (used <= 0) checks with (used == 0) checks to remove unnecessary checks.
Replace (used != 0) checks with (used) checks to avoid extra processor work.
When incrementing and decrementing buffer, do not report used as an error, instead just return F_data_not.
Shorten pre_allocate_size to size_allocate.
Stop setting F_unterminated_group_stop and F_unterminated_group_eos as warnings.
This was not done consistently.
Rename buffer to destination for FSS write functions.
At some point in the past I was wanting to use while loops more exclusively.
After some some significant time and consideration, I have decided to abandon that idea.
Use for loops where they make more sense over while loops.
Kevin Day [Mon, 6 Jul 2020 03:49:56 +0000 (22:49 -0500)]
Progress: featureless make.
Update appropriate f_utf_ functions.
Implement "define" and "parameter" IKI variables as per IKI-0002.
These variables will be substituted with settings value if found or removed if not found.
This expansion will happen before performing operation.
rename fake_make_load_setting() to fake_make_load_fakefile() to be more explicit.
Kevin Day [Sun, 5 Jul 2020 23:59:09 +0000 (18:59 -0500)]
Update: Unicode related specification adjustments, also perform minor fixes and cleanups.
The Unicode includes some characters that violate some of the expectations of the FSS and IKI standards.
To that end, provide non-strict modes to allow for the FLL based adjustments of these standards.
There are some vertical line like characters that act as punctuation connectors.
Because they connect vertically instead of horizontally, they violate part of the inteded design of FSS and IKI where everything is on a per-line basis.
These Unicode vertical line like punctuation connectors are not to be considered punctuation connectors for the purposes of FSS and IKI processing.
There are some non-printing characters, such as invisible plus.
This is a punctuation character that is also a zero-width character.
For the purposes of FSS and IKI, these characters must not be considered punctuation characters.
Any other zero-width characters are to be considered zero-width and are not to be used for syntax and similar.
(Combining characters are a different beast to be addessed at a later time.)
Further clarify how certain Unicode characters are to be handled, in general.
Unicode dash characters are now being limited in being interpreted as a dash for the purposes of connecting words.
To this end, only the Unicode hpyhens are the allowed dash-like characters (with the ASCII dash being allowed as well for that matter).
Kevin Day [Sun, 5 Jul 2020 22:33:03 +0000 (17:33 -0500)]
Update: improve IKI syntax rules.
This is a relaxation on the specification where previously only whitespace was required to exist before a valid variable name.
This allows punctuation to exist before the variable name.
Content such as 'A:"b"' may now be specified where A is the variable name (notice the single-quote before A which would previously prevent A from being recognized as a variable name).
Kevin Day [Fri, 3 Jul 2020 03:58:27 +0000 (22:58 -0500)]
Bugfix: FSS-0000 and FSS-0001 are handling not explicitly follow standard for quote delimit requirements.
In the case where a quote is provided that could potentially be a valid quote, that quote should only be considered a quote (including for delimited purposes) if a valid whitespace, end of string, or end of line is reached.
The code needs to check after the closing quote to confirm that it is actually a valid closing quote before acting on it.
Update the FSS specification to make this behavior more explicitly clear.
Kevin Day [Thu, 2 Jul 2020 03:36:33 +0000 (22:36 -0500)]
Progress: UTF-8
More UTF-8 improvements.
The Punctuation is complete as far as the "Other" punctuation category is concerned.
(Apparently, there are more groups/sets to do!)
Create a new whitespace functions (is_whitespace_other()) to handle non-whitespace whitespace.
So far, this is only a single character: Ogham space mark ( ).
This Ogham space mark is not a whitespace, it is a graph character that is considered a whitespace.
Add some @todos.
Remove invalid whitespace character().
Add is_digit() vs is_numeric().
The traditional alnum() is referring to alphabet and digit.
With UTF-8 there are non-digit numbers, which can be classified as "numeric".
Therefore, the is_digit() functions are added and intended to be used for the traditional behavior.
The is_numeric() are intended to handle any numeric character.
The is_word() tests are for digits and not for numeric.
Kevin Day [Mon, 29 Jun 2020 04:25:14 +0000 (23:25 -0500)]
Bugfix: IKI Read should not always print content on substitution in whole mode.
When printing in whole mode, it is only printing the content, even when --literal is passed.
What should happen is that when --literal is passed, the entire variable should be printed, even when substitution is used.
The substitution will still occur.
Kevin Day [Mon, 29 Jun 2020 00:09:25 +0000 (19:09 -0500)]
Update: finish implementing IKI Read program.
The resulting design made --expand redundant, so it is removed.
Substitution is implemented and works independent of the --name parameter.
Added a new macro to the FLL project called f_macro_memory_structure_macro_increment().
Too often do I have to create those structures, so here is a macro to perform the increment.
I started to update the rest of the project and then decided not to spend time on that right now.
This, therefore, primarily focuses on getting the macro working in IKI.
IKI Read is now considered usable, except for the UTF-8 support.
The UTF-8 support is dependent on updating a ton of UTF-8 codes.
This will be done at a later time.
The IKI Read program will eventually need to be reviewed for mistakes and cleanups.
Kevin Day [Sat, 27 Jun 2020 23:17:45 +0000 (18:17 -0500)]
Bugfix: when --line is used, lengths passed buffer size aren't handled.
data->line is not a position, do not use it as a position (if (data->line < data->buffer.used) { is doing this).
If the line is exceeds the buffer, return f_data_not and properly handle that return case.
For --total, this means print 0.
For just about everything else, print nothing.
Kevin Day [Sat, 27 Jun 2020 23:03:46 +0000 (18:03 -0500)]
Bugfix: delimit placeholder not being applied in all cases
The final exist should apply any delimits.
When determining the remaining width, make sure that the range or buffer has not yet been exceeded.
Not doing this resulted in an exit with an F_parameter error, further resulting in no delimits being applied.
Kevin Day [Sat, 27 Jun 2020 04:33:04 +0000 (23:33 -0500)]
Progress: IKI Read.
Implement the basics of raw mode.
Raw mode should instead work in conjunction with the content, literal, and object modes so redesign accordingly.
Add some additional documentation to the help section.
Expand and Substitute are mostly ignored thus far and will be implemented at a later time.
The substitution will be copied from the argv as-is, so plan on using static strings instead of dynamic strings.
There will be function argument cleaning to do once I determine how Expand and Substitute will be implemented.
Until then, the arguments are fine as-is.
Kevin Day [Fri, 26 Jun 2020 02:54:12 +0000 (21:54 -0500)]
Bugfix: read overflows the per character scanning
There is a logic mistake where the buffer is incremented too many times before the next loop.
This results in skipping some of the characters in the file and if those skipped characters are part of a vocabulary and content, then it will not be detected.
Kevin Day [Fri, 26 Jun 2020 02:51:59 +0000 (21:51 -0500)]
Cleanup: reserve IKI-0000 and update documentation
The standard IKI-0000 is now to be used to represent anything that follows the IKI standard without explicitly descrribing what its vocabulary is.
All other IKI standards (IKI-0001 and greater) are to be used to represent a specific set of vocabulary terms.
Kevin Day [Thu, 25 Jun 2020 04:00:41 +0000 (23:00 -0500)]
Update: dependencies changes, project level changes, and includes changes.
Review the current dependencies and fix them.
There are some cases where some dependencies are not necessary, such as f_utf.
There are some cases where some dependencies are missing (or newly added).
f_conversion split into f_conversion and fl_conversion.
Update dependencies accordingly.
For simplicity, include the core level_0 project headers, except in a small number of cases.
Add fss header comment in dependencies files.
Add comments in *_array.h headers for level_0 core projects.
Add missing header in f_string.
Replace _di_level_0_parameter_checking_ with _di_level_1_parameter_checking_ where incorrect.
The fl_fss identify functions should instead be in the fll_fss project.
Make sure the IKI projects are in the bootstrap example script.
Remove the fl_console_parameter_to_number_unsigned() and similar functions.
These are just wrappers to another call and is therefore a waste.
Instead, just call the fl_conversion_string_to_number_unsigned() and similar functions.
Kevin Day [Wed, 24 Jun 2020 05:14:04 +0000 (00:14 -0500)]
Bugfix: conversion needs to properly handle '0x' and similar
Invalid Parameter happens when there is no following number after the x in '0x'.
This is because the start and stop range were not being checked after applying the offset.
Kevin Day [Tue, 23 Jun 2020 03:54:54 +0000 (22:54 -0500)]
Progress: Featureless Make.
Begin working on integrating IKI into Fake.
I took advantage of the current code to do some testing with IKI and left some of the testing code for now.
In addition to IKI, I am thinking that I need to add another project for exploding a string into separate arguments with support for escapable single and double quotes.
Kevin Day [Mon, 22 Jun 2020 03:01:52 +0000 (22:01 -0500)]
Feature: add fl_iki project.
This provides fl_iki_read, a function for identifying all iki within a given range.
Whereas the f_iki_read function will only identify the first valid iki variable found within the given range.
Kevin Day [Mon, 22 Jun 2020 02:58:21 +0000 (21:58 -0500)]
Update: add missing dependencies, changes to iki
In a previous commit I seem to have forgotten to include the build settings changes.
Fix some minor mistakes in iki.
In particular, I forgot to increment the range->start after identifying where content would begin (which prevented it from working at all).
Add f_iki_variable to store the range for the entire variable.
This will allow for easy removal when substituting (which is one of the design intentions, so I need to cover this use case!).
Kevin Day [Sun, 21 Jun 2020 23:43:01 +0000 (18:43 -0500)]
Update: UTF-8 improvements
Move some FSS UTF-8 processing code into UTF-8 as it should be more generic.
Add more UTF-8 functions (or at least the stubs).
Begin adding emoji, which is all over the place.
Staring at so many codes, it will be easy to make a mistake.
After all of the UTF-8 functions are believed to be fully implemented an extensive review needs to happen.
Kevin Day [Sun, 21 Jun 2020 01:03:42 +0000 (20:03 -0500)]
Progress: organize headers, simplify fss structures, initial IKI support
Move more headers into the secondary structures to simplify the main header, which will now more often be just for function definitions.
The FSS structures can use a more common base to better allow integration with separate projects that utilize the same basic design.
This is important for IKI project.
Casting to/from fss objects and contents to their respective ranges should now be possible.
Begin adding the IKI support, which is a new FSS standard focused on a simpler/smaller Wiki design.
The goal is not to handle presentation but instead to provide very simple context on certain text blocks based on some vocabulary.
A very common example would be sending a plain text e-mail with a URL in it, the URL could be encapsulated in an IKI vocabulary object to tell both the human and the computer that this is a URL.
For a URL, this is very useful in that no part of the URL need be escaped, except for the closing quote and its respective delimiter.
Example iki URL: url:"http://www.example.com/this does not exist/".
As opposed to http://www.example.com/this%20does%20not%20exist/.
The first case is easier for a human while still remaining simple enough for a machine.
The IKI syntax, at least in its most basic form, is not meant to be a full WIKI formatter.
While this may be accomplished with a sufficiently large enough vocabulary, this seems unnecessary at this time.
The IKI syntax standard itself is yet to be ironed out and ill be updated overtime until I am satisfied.
Kevin Day [Sat, 20 Jun 2020 19:08:03 +0000 (14:08 -0500)]
Bugfix: revert slash improvements in FSS object read
Oops.
When relocating this I completely forgot that an object starting with something other than a double quote or single quote can never be a quoted object.
Therefore only the first slash needs to be delimited.
I need to update the FSS documentation to avoid this oops again.
This will be tacked on to the todo list for when I do my extensive review of the FSS processing code.
Kevin Day [Sat, 20 Jun 2020 16:46:24 +0000 (11:46 -0500)]
Progress: update FSS code, including zero-width updates
Update FSS to consider the recent zero-width improvements.
Making the changes reminded me on how I still need to go through and clean the FSS code up.
As noted in previous commits, the changes into supporting UTF really put the code into an unideal and not well tested state.
This needs extensive review and fixes.
To that end, I felt compelled to fix problems and perform cleanups that I observed.
At least to a small extent.
The basic and extended object reads are identical so move the code into a shared private function.
One behavioral change is to better support fail-through practice.
For FSS Extended, the contents are always terminated by a newline.
However, if there is an unterminated quote, then the entire process would previously fail.
This time, do no accept content for that line and continue on to next content.
Kevin Day [Fri, 19 Jun 2020 20:37:16 +0000 (15:37 -0500)]
Update: add additional f_fss functions and update documentation
Add line counting functions.
Remove unnecessary checks against negative values for unsigned numbers.
(They are theoretically necessary in that if the type was overwritten to a signed type, but for now just ignore that case.)
Finish adding or updating the *_together functions.
Review and update the documentation.
Kevin Day [Thu, 18 Jun 2020 05:27:01 +0000 (00:27 -0500)]
Progress: featureless make
Continue development in featureless make, implementing support for "make" operation.
This expands the f_string project to support additional structures.
This expands the fll_fss project to utilize these additional structures.
I have reviewed and redesigned some of the fll_fss project functions.
The fll_fss project function redesign is incomplete and not fully tested as of this commit.
(I plan on reviewing and updating the *_together functions.)
The current fakefile is a stub/example for testing and will be updated to a correct default fakefile once the "make" operation is complete.
The plan is to have the fakefile provide an example for building th featureless make project using the fakefile.
This would mean that the featuereless make project can be built by the featureless make project using either the "build" or the "make" operation.
Kevin Day [Mon, 15 Jun 2020 04:58:02 +0000 (23:58 -0500)]
Feature: add fss_named and fss_set structures and re-organize the fss headers
Provide fss_named and fss_set structures (and their respective array structures).
These are useful when using FSS types nested within another set (not to be confused with fss_nest which is for a single type with recursion).
There are just too many things in fss.h.
Break it out into multiple headers that are auto-included.
Kevin Day [Mon, 15 Jun 2020 04:54:58 +0000 (23:54 -0500)]
Feature: add macro initializers for static string and string ranges
Creating defines with present lengths and them using these in static strings, sometimes with ranges, has become more common.
Provide the macros to make this simpler and ever be more usable for creating constant static strings and constant string ranges.
Kevin Day [Fri, 12 Jun 2020 02:49:46 +0000 (21:49 -0500)]
Progress: UTF-8 file write should handle case where expanded character is larger than write max
When the 4-byte wide UTF-8 character is expanded into potentally 4 1-byte wide UTF-8 characters, the available bytes being expanded need to fit in the output buffer.
Kevin Day [Mon, 8 Jun 2020 23:29:15 +0000 (18:29 -0500)]
Update: fl_serialized to f_serialize, renaming f_serialized to f_serialize
There is nothing forcing the current functions in fl_serialized from being in f_serialize.
Renamed f_serialized to f_serialize to be more consistent with the function names.
To that end, I have renamed the functions like f_unserialize* to f_serialize_un.
I have been unable to come up with a better name that is also reasonably short.
Kevin Day [Mon, 8 Jun 2020 04:34:49 +0000 (23:34 -0500)]
Update: documentation cleanups, fix observed problems, retun F_data_not without error bit
Organize the return codes.
Fix any probles in the commend documentation that I happened to notice.
This is not complete, there will be follow up commits to address more cleanups.
I noticed that there were some public directory functions incorrectly have 'private_' in their name.
The conversion functions should no longer consider no data an error.
Just report that there was nothing to do.
Kevin Day [Mon, 8 Jun 2020 02:16:51 +0000 (21:16 -0500)]
Update: documentation, also use fss-0002
Fix the documentation to clarify that the --work paramter is, in fact, more different than --prefix parameter.
Update the tree examples to show both the build directory tree and work directory tree.
Switch to using the fss-0002 format for all of the documentation files and specification files.
Kevin Day [Sun, 7 Jun 2020 21:40:24 +0000 (16:40 -0500)]
Update: implement --process parameter
Looks like I forgot to implement this.
The --process parameter is a process name that is prepended to stage file names for the purpose of granting the caller additional flexibility.
Kevin Day [Sun, 7 Jun 2020 19:05:29 +0000 (14:05 -0500)]
Update: bootstrap dependency functionality, fix missing dependencies, also ensure level and monolithic
It seems I wrote a dependency generator to make my life easier and then I went and forgot about it.
Update this to work with the more recent changes to the build settings structure.
While I am at it, generate all of the dependencies and save the generated dependencies.
This exposed a typoe in the dependencies file, now fixed.
Might as well make sure the level and monolithic dependencies are generated as well.
Kevin Day [Sun, 7 Jun 2020 14:07:06 +0000 (09:07 -0500)]
Update: support copying documents, licenses, and specifications per project and do some cleanups
Individual projects, specifically programs (level_3), may have custom documentation, licenses, and specifications.
(With custom licenses being least likely because this project in its entirety is under lgpl.)
Add support to all projects just for good practice.
Cleanup ordering of function names and add spaces before function instantiation open brace.
Kevin Day [Sun, 7 Jun 2020 03:30:17 +0000 (22:30 -0500)]
Update: Featureless Make needs to handle the -d/--define parameter
Rename the parameter from defines to define, because it acts on only a single parameter at a time.
The -d must be specified multiple times to get multiple defines.
When -d/--define is specified, the defines provided by the settings file get ignored.
Continue to pass the defines from the console parameters to the process scripts.