fix(template-engine): removed gsl, added pyratemp

As GSL failed in my first attempt to get the example program going,
it might be better to try something else before too much time is spend.

Fortunately, pyratemp **seems** to be something usable, and even if not,
it might be possible to make it usable as it's just a 'simple'
python script that I might be able to understand, if need be.
This commit is contained in:
Sebastian Thiel
2015-03-01 13:24:01 +01:00
parent f2ca8c3fb7
commit e06738a7bd
8 changed files with 244 additions and 310 deletions

View File

@@ -3,30 +3,24 @@
include Makefile.helpers
PYTHON = python2.7
CONVERT = $(PYTHON) ./etc/bin/json2xml.py
GSL = ./etc/bin/gsl_$(OS)-$(ARCH) -q
TPL = etc/bin/pyratemp.py
API_DEPS = .api.deps
API_SHARED_XML = ./etc/api/shared.xml
API_SHARED_INFO = ./etc/api/shared.yml
API_JSON_FILES = $(shell find ./etc -type f -name '*-api.json')
API_XML_FILES = $(patsubst %.json,%.xml,$(API_JSON_FILES))
help:
$(info Programs)
$(info ----> GSL: '$(GSL)')
$(info ----> json2xml: '$(CONVERT)')
$(info ----> templat engine: '$(TPL)')
$(info )
$(info Targets)
$(info help - print this help)
$(info json-to-xml - convert json API files to xml for consumption by GSL)
$(info api-deps - generate a file to tell make what API file dependencies will be)
json-to-xml: $(API_XML_FILES)
$(API_DEPS): $(API_SHARED_XML)
$(GSL) -script:src/gsl/deps.gsl $(API_SHARED_XML)
%.xml: %.json
$(CONVERT) --pretty -o $@ < $<
$(API_DEPS): $(API_SHARED_INFO)
$(TPL) -f $(API_SHARED_INFO) -d DEP_FILE=$@
api-deps: $(API_DEPS)

View File

@@ -21,13 +21,10 @@ The license of everything not explicitly under a different license are licensed
What follows is a list of other material that is licensed differently.
* **./etc/bin/json2xml.py** is licensed like MIT, as shown in the header of the file. See original source [on github][html2json].
* **./etc/bin/gsl_\*** is licensed under [GNU GPL][imatix-copying]. The source code is [on github][gsl].
* **./etc/bin/pyratemp.py** is licensed under MIT, as shown in [the header][pyratemp-header] of the file.
* **./etc/api/\*\*/*.json** are licensed under a [MIT-like google license][google-lic].
[oauth]: https://crates.io/crates/yup-oauth2
[html2json]: https://github.com/hay/xml2json
[imatix-copying]: https://github.com/imatix/gsl/blob/master/COPYING
[gsl]: https://github.com/imatix/gsl
[pyratemp-header]: https://github.com/Byron/youtube-rs/blob/master/etc/bin/pyratemp.py
[google-lic]: https://github.com/google/google-api-go-client/blob/master/LICENSE

Binary file not shown.

Binary file not shown.

View File

@@ -1,285 +0,0 @@
#!/usr/bin/env python
"""
Copyright (C) 2010-2013, Hay Kranen
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
"""xml2json.py Convert XML to JSON
Relies on ElementTree for the XML parsing. This is based on
pesterfish.py but uses a different XML->JSON mapping.
The XML->JSON mapping is described at
http://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html
Rewritten to a command line utility by Hay Kranen < github.com/hay > with
contributions from George Hamilton (gmh04) and Dan Brown (jdanbrown)
XML JSON
<e/> "e": null
<e>text</e> "e": "text"
<e name="value" /> "e": { "@name": "value" }
<e name="value">text</e> "e": { "@name": "value", "#text": "text" }
<e> <a>text</a ><b>text</b> </e> "e": { "a": "text", "b": "text" }
<e> <a>text</a> <a>text</a> </e> "e": { "a": ["text", "text"] }
<e> text <a>text</a> </e> "e": { "#text": "text", "a": "text" }
This is very similar to the mapping used for Yahoo Web Services
(http://developer.yahoo.com/common/json.html#xml).
This is a mess in that it is so unpredictable -- it requires lots of testing
(e.g. to see if values are lists or strings or dictionaries). For use
in Python this could be vastly cleaner. Think about whether the internal
form can be more self-consistent while maintaining good external
characteristics for the JSON.
Look at the Yahoo version closely to see how it works. Maybe can adopt
that completely if it makes more sense...
R. White, 2006 November 6
"""
import json
import optparse
import sys
import xml.etree.cElementTree as ET
from xml.dom import minidom
def strip_tag(tag):
strip_ns_tag = tag
split_array = tag.split('}')
if len(split_array) > 1:
strip_ns_tag = split_array[1]
tag = strip_ns_tag
return tag
def elem_to_internal(elem, strip_ns=1, strip=1):
"""Convert an Element into an internal dictionary (not JSON!)."""
d = {}
elem_tag = elem.tag
if strip_ns:
elem_tag = strip_tag(elem.tag)
else:
for key, value in list(elem.attrib.items()):
d['@' + key] = value
# loop over subelements to merge them
for subelem in elem:
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
tag = subelem.tag
if strip_ns:
tag = strip_tag(subelem.tag)
value = v[tag]
try:
# add to existing list for this tag
d[tag].append(value)
except AttributeError:
# turn existing entry into a list
d[tag] = [d[tag], value]
except KeyError:
# add a new non-list entry
d[tag] = value
text = elem.text
tail = elem.tail
if strip:
# ignore leading and trailing whitespace
if text:
text = text.strip()
if tail:
tail = tail.strip()
if tail:
d['#tail'] = tail
if d:
# use #text element if other attributes exist
if text:
d["#text"] = text
else:
# text is the value if no attributes
d = text or None
return {elem_tag: d}
def internal_to_elem(pfsh, factory=ET.Element):
"""Convert an internal dictionary (not JSON!) into an Element.
Whatever Element implementation we could import will be
used by default; if you want to use something else, pass the
Element class as the factory parameter.
"""
attribs = {}
text = None
tail = None
sublist = []
tags = list(pfsh.keys())
if len(tags) > 1:
raise ValueError("Illegal structure with multiple tags: %s" % tag)
tag = tags[0]
value = pfsh[tag]
def sani_value(v):
if v is None:
return v
return unicode(v)
# we santize values automatically
# $ref -> _ref
if tag.startswith('$'):
tag = '_' + tag[1:]
if ':' in tag:
assert isinstance(value, dict), "hardcoded requirement: value must be dict for us to sanitize tag"
tk = 'value'
assert tk not in value
value[tk] = tag
tag = 'item'
if isinstance(value, dict):
for k, v in value.items():
if k[:1] == "@":
attribs[k[1:]] = v
elif k == "#text":
text = v
elif k == "#tail":
tail = v
elif isinstance(v, list):
for v2 in v:
sublist.append(internal_to_elem({k: v2}, factory=factory))
else:
sublist.append(internal_to_elem({k: v}, factory=factory))
else:
text = value
e = factory(tag, attribs)
for sub in sublist:
e.append(sub)
e.text = sani_value(text)
e.tail = sani_value(tail)
return e
def elem2json(elem, options, strip_ns=1, strip=1):
"""Convert an ElementTree or Element into a JSON string."""
if hasattr(elem, 'getroot'):
elem = elem.getroot()
if options.pretty:
return json.dumps(elem_to_internal(elem, strip_ns=strip_ns, strip=strip), sort_keys=True, indent=4, separators=(',', ': '))
else:
return json.dumps(elem_to_internal(elem, strip_ns=strip_ns, strip=strip))
def json2elem(json_data, factory=ET.Element):
"""Convert a JSON string into an Element.
Whatever Element implementation we could import will be used by
default; if you want to use something else, pass the Element class
as the factory parameter.
"""
return internal_to_elem(json.loads(json_data), factory)
def xml2json(xmlstring, options, strip_ns=1, strip=1):
"""Convert an XML string into a JSON string."""
elem = ET.fromstring(xmlstring)
return elem2json(elem, options, strip_ns=strip_ns, strip=strip)
def json2xml(json_data, options, factory=ET.Element):
"""Convert a JSON string into an XML string.
Whatever Element implementation we could import will be used by
default; if you want to use something else, pass the Element class
as the factory parameter.
"""
if not isinstance(json_data, dict):
json_data = json.loads(json_data)
if len(json_data.keys()) > 1:
json_data = dict(root = json_data)
elem = internal_to_elem(json_data, factory)
xml_str = ET.tostring(elem)
if options.pretty:
xml_str = minidom.parseString(xml_str).toprettyxml(indent='\t')
return xml_str
def main():
p = optparse.OptionParser(
description='Converts XML to JSON or the other way around. Reads from standard input by default, or from file if given.',
prog='xml2json',
usage='%prog -t xml2json -o file.json [file]'
)
p.add_option('--type', '-t', help="'xml2json' or 'json2xml'", default="json2xml")
p.add_option('--out', '-o', help="Write to OUT instead of stdout")
p.add_option(
'--strip_text', action="store_true",
dest="strip_text", help="Strip text for xml2json")
p.add_option(
'--pretty', action="store_true",
dest="pretty", help="Format JSON/HTML output so it is easier to read")
p.add_option(
'--strip_namespace', action="store_true",
dest="strip_ns", help="Strip namespace for xml2json")
p.add_option(
'--strip_newlines', action="store_true",
dest="strip_nl", help="Strip newlines for xml2json")
options, arguments = p.parse_args()
inputstream = sys.stdin
if len(arguments) == 1:
try:
inputstream = open(arguments[0])
except:
sys.stderr.write("Problem reading '{0}'\n".format(arguments[0]))
p.print_help()
sys.exit(-1)
input = inputstream.read()
strip = 0
strip_ns = 0
if options.strip_text:
strip = 1
if options.strip_ns:
strip_ns = 1
if options.strip_nl:
input = input.replace('\n', '').replace('\r','')
if (options.type == "xml2json"):
out = xml2json(input, options, strip_ns, strip)
else:
out = json2xml(input, options)
if (options.out):
file = open(options.out, 'wb')
file.write(out.encode('utf-8'))
file.close()
else:
print(out.encode('utf-8'))
if __name__ == "__main__":
main()

237
etc/bin/pyratemp.py Executable file
View File

@@ -0,0 +1,237 @@
#!/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
# -*- coding: ascii -*-
"""
Commandline-tool for pyratemp.
- check template for syntax-errors
- render templates with data and print the result in utf-8.
Errors and help-messages are written to stderr, resulting data to stdout.
Data can be read as key-value-pairs from the command-line and/or from
JSON- and YAML-files. Additionally, ``date`` and ``mtime_CCMMYYDD``
are set to the current date as "%Y-%m-%d".
By default, HTML-escaping is used for "*.htm" and "*.html" and
LaTeX-escaping for "*.tex".
Exit-codes:
- 0: ok
- 1: some Python-modules are missing (import error)
- 2: --help / usage printed
- 3: invalid command-line options
- 10: template syntax-error / parse error
- 20: datafile error / cannot load data
- 30: render error
:Version: 0.3.2
:Usage:
see USAGE or "pyratemp_tool.py --help"
:Requires: Python >= 2.6 / 3.x, pyratemp, (optional: yaml)
:Author: Roland Koebler (rk at simple-is-better dot org)
:Copyright: Roland Koebler
:License: MIT/X11-like, see __license__
:RCS: $Id: pyratemp_tool.py,v 1.16 2013/09/17 07:45:04 rk Exp $
"""
from __future__ import unicode_literals
from __future__ import print_function
__version__ = "0.3.2"
__author__ = "Roland Koebler <rk at simple-is-better dot org>"
__license__ = """Copyright (c) 2007-2013 by Roland Koebler
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE."""
#-----------------------------------------
USAGE = """pyratemp_tool.py [-s] <-d NAME=VALUE> <-f DATAFILE [-N NAME] [-n NR_OF_ENTRY]> [--xml] TEMPLATEFILES
-s syntax-check only (don't render the template)
-d define variables (these also override the values from files)
-f use variables from a JSON/YAML file
-n use nth entry of the JSON/YAML-file
(JSON: n-th element of the root-array, YAML: n-th entry)
-N namespace for variables from the JSON/YAML file
--xml encode output as ASCII+xmlcharrefreplace (instead of utf-8)
"""
#-----------------------------------------
import os, sys, getopt, time
try:
import pyratemp
except ImportError:
print("ERROR: Python-module 'pyratemp' not found.", file=sys.stderr)
sys.exit(1)
#-----------------------------------------
def parse(template_name):
"""Parse template + set encoding according to filename-extension.
:Returns: the parsed template
"""
ext = os.path.splitext(template_name)[1]
if ext == ".htm" or ext == ".html":
t = pyratemp.Template(filename=template_name, escape=pyratemp.HTML)
elif ext == ".tex":
t = pyratemp.Template(filename=template_name, escape=pyratemp.LATEX)
else:
t = pyratemp.Template(filename=template_name)
return t
#----------------------
def load_data(datafiles):
"""Load data from data-files using either 'json' or 'yaml'.
:Parameters:
- datafiles: [ [filename, nr_of_entry, namespace], ...]
:Returns: read data (dict)
:Raises: ImportError, ValueError
"""
imported_json = False
imported_yaml = False
mydata = {}
for filename, n, namespace in datafiles:
if filename[-5:].lower() == ".json":
if not imported_json:
try:
import simplejson as json
except ImportError:
import json
imported_json = True
try:
myjson = json.load(open(filename, 'r'))
if n != -1:
myjson = myjson[n]
if namespace is None:
mydata.update(myjson)
else:
mydata.update({namespace: myjson})
except ValueError as err:
raise ValueError("Invalid JSON in file '%s'. (%s)" % (filename, str(err)))
elif filename[-5:].lower() == ".yaml":
if not imported_yaml:
import yaml
imported_yaml = True
if n == -1:
n = 0
myyaml = yaml.load_all(open(filename, 'r'))
if namespace is not None:
mydata.update({namespace: list(myyaml)[n]})
else:
mydata.update(list(myyaml)[n])
else:
raise ValueError("Invalid data-file '%s', must be .json or .yaml" % filename)
return mydata
#-----------------------------------------
if __name__ == "__main__":
# parse parameters
try:
opt_list, files = getopt.getopt(sys.argv[1:], "sd:f:n:N:h", ("help", "xml"))
except getopt.GetoptError as err:
print("ERROR: Invalid option. (%s)" % err, file=sys.stderr)
sys.exit(3)
render = True
template_name = ""
namevals = {}
datafiles = [] #[ [filename, nr_of_entry], ...]
output_xml = False
for key, value in opt_list:
if "-h" == key or "--help" == key:
print(USAGE, file=sys.stderr)
sys.exit(2)
elif "-s" == key:
render = False
elif "-d" == key:
(name, value) = value.split("=", 1)
namevals[name] = value
elif "-f" == key:
datafiles.append([value, -1, None])
elif "-n" == key:
if not datafiles:
print("ERROR: -n only allowed after -f.", file=sys.stderr)
sys.exit(3)
datafiles[-1][1] = int(value)
elif "-N" == key:
if not datafiles:
print("ERROR: -N only allowed after -f.", file=sys.stderr)
sys.exit(3)
datafiles[-1][2] = value
elif "--xml" in key:
output_xml = True
if not files:
print(USAGE, file=sys.stderr)
sys.exit(2)
for f in files:
if f == "--":
break
elif f[0] == "-":
print("ERROR: Invalid order of parameters. (%s)" % f, file=sys.stderr)
sys.exit(3)
# template
for template_name in files:
# parse + syntax-check
try:
t = parse(template_name)
except pyratemp.TemplateSyntaxError as err:
print("file '%s':" % template_name, file=sys.stderr)
print(" TemplateSyntaxError:", str(err), file=sys.stderr)
sys.exit(10)
if render:
# load data
try:
filedata = load_data(datafiles)
except ImportError as err:
print("ImportError/missing Python-module:", str(err), file=sys.stderr)
sys.exit(1)
except ValueError as err:
print("Datafile error:", str(err), file=sys.stderr)
sys.exit(20)
localtime = time.localtime()
data = {
'mtime_CCYYMMDD':time.strftime("%Y-%m-%d",localtime),
'date' :time.strftime("%Y-%m-%d",localtime),
}
data.update(filedata)
data.update(namevals)
data = pyratemp.dictkeyclean(data)
# render
try:
if output_xml:
result = t(**data).encode("ascii", "xmlcharrefreplace")
else:
result = t(**data).encode("utf-8")
os.write(sys.stdout.fileno(), result)
except pyratemp.TemplateRenderError as err:
print("file '%s':\n" % template_name, file=sys.stderr)
print(" TemplateRenderError:", str(err), file=sys.stderr)
sys.exit(30)
#-----------------------------------------

View File

@@ -1,9 +0,0 @@
.template 0
amount = 1000
year = 2006
while year < 2026
amount = amount * 1.05
year = year + 1
endwhile
echo amount
.endtemplate