29: Optimally Use LibreOffice as a Files Converter (a Python Implementation)|T.B.P.

<The previous article in this series | The table of contents of this series | The next article in this series>

Between OpenDocument/Microsoft Office/etc. formats and to the PDF/CSV format, fast and versatile with the document possibly tweaked. A workable sample.

Topics

About: UNO (Universal Network Objects)
About: LibreOffice
About: Apache OpenOffice
About: Python programming language

The table of contents of this article

Starting Context
Target Context
Orientation
Main Body
0: A Note: 'soffice --convert-to' Will Be Never Used
1: An Implementation of the File Conversion Logic with the UNO API
2: On Properties for Opening or Storing
3: On Tweaking the Document
4: Here Is a Workable Sample
4-1: Building the Sample Project
4-2: Executing the Sample Program
4-3: Understanding the Sample Code

Starting Context

The reader has a basic knowledge on Python programming.
The reader has a knowledge on what UNO is and how it is related to LibreOffice or Apache OpenOffice.
The reader has a knowledge on the basic elements of UNO and the terminology used for them in this series.
The reader has built an environment for developing UNO programs in Linux or Windows, according to a previous article (here for Linux and here for Windows).
The reader has a knowledge on how to make any LibreOffice or Apache OpenOffice instance a UNO server.
The reader has a knowledge on how to create a Python external UNO client in a just-connecting way or in a connection-aware way.
The reader has a knowledge on the concept of how to optimally use LibreOffice or Apache OpenOffice as a files converter.

Target Context

The reader will understand how to implement file conversion directly using the UNO API in Python, and will have a workable sample.

Orientation

Main Body

Stage Direction
Here are Special-Student-7, Lenard (a Python programmer), and Jane (a Python programmer) in front of a computer.

0: A Note: 'soffice --convert-to' Will Be Never Used

Special-Student-7
While it is clearly stated that the concept part of this article is supposed to have been read, I can guess that a goodly portion of the readers have not and will not read it, and considering the fact that many people are fixated on using 'soffice --convert-to', I feel it appropriate to remind you again that 'soffice --convert-to' will be never used here.

The reason is elaborated in the concept part.

1: An Implementation of the File Conversion Logic with the UNO API

Special-Student-7
Supposing that you already know what you need to know from the concept part of this article, let us immediately see a Python implementation of the file conversion logic with the UNO API, where 'l_underlyingRemoteUnoObjectsContextInXComponentContext' is a UNO objects context to the LibreOffice or Apache OpenOffice instance, 'l_originalFileUrl' is the URL of the original file, 'l_targetFileUrl' is the URL of the target file, and 'l_fileStoringFilterName' is the file-storing-filter name.

@Python Source Code

import sys
from typing import List
from typing import Match
from typing import Optional
from typing import Pattern
from typing import cast
import re
import uno
from com.sun.star.beans import PropertyValue
from com.sun.star.frame import XDispatchProvider
from com.sun.star.frame import XStorable2
from com.sun.star.frame import XSynchronousDispatch
from com.sun.star.uno import XComponentContext
from com.sun.star.uno import XInterface
from com.sun.star.util import URL as com_sun_star_util_URL
from com.sun.star.util import XCloseable
~

class Test1Test:
	c_urlRegularExpression: Pattern  = re.compile ("(.*:)(.*)")
	
	@staticmethod
	def getUrlInURL (a_url: str) -> com_sun_star_util_URL:
		l_urlInURL: com_sun_star_util_URL = com_sun_star_util_URL ()
		l_urlInURL.Complete = a_url
		l_urlInURL.Main = l_urlInURL.Complete
		l_regularExpressionMatch: Optional [Match] = Test1Test.c_urlRegularExpression.match (l_urlInURL.Complete, 0)
		if l_regularExpressionMatch is not None:
			l_urlInURL.Protocol = l_regularExpressionMatch.groups () [0]
			l_urlInURL.User = ""
			l_urlInURL.Password = ""
			l_urlInURL.Server = ""
			l_urlInURL.Port = 0
			l_urlInURL.Path = l_regularExpressionMatch.groups () [1]
			l_urlInURL.Name = ""
			l_urlInURL.Arguments = ""
			l_urlInURL.Mark = ""
		return l_urlInURL
	
	@staticmethod
	def main (a_arguments: List [str]) -> None:
		~
				l_originalFileUrl: str = a_arguments [2];
				l_targetFileUrl: str = a_arguments [3];
				l_fileStoringFilterName: str = a_arguments [4];
				
				l_underlyingUnoDesktopInXDispatchProvider: XDispatchProvider = cast (XDispatchProvider, cast (XInterface, l_underlyingRemoteUnoObjectsContextInXComponentContext.getValueByName ("/singletons/com.sun.star.frame.theDesktop")))
				l_underlyingFileOpeningUnoDispatcherInXSynchronousDispatch: XSynchronousDispatch = cast (XSynchronousDispatch, l_underlyingUnoDesktopInXDispatchProvider.queryDispatch (Test1Test.getUrlInURL ("file:///"), "_blank", -1))
				
				l_originalFileUrlInURL: com_sun_star_util_URL = Test1Test.getUrlInURL (l_originalFileUrl)
				l_unoDocumentOpeningProperties: List [PropertyValue] = [None, None, None, None]
				l_unoDocumentOpeningProperties [0] = PropertyValue ("ReadOnly", -1, True, 0);
				l_unoDocumentOpeningProperties [1] = PropertyValue ("Hidden", -1, True, 0);
				l_unoDocumentOpeningProperties [2] = PropertyValue ("OpenNewView", -1, True, 0);
				l_unoDocumentOpeningProperties [3] = PropertyValue ("Silent", -1, True, 0);
				l_unoDocumentStoringProperties: List [PropertyValue] = [None, None]
				l_unoDocumentStoringProperties [0] = PropertyValue ("FilterName", -1, l_fileStoringFilterName, 0)
				l_unoDocumentStoringProperties [1] = PropertyValue ("Overwrite", -1, True, 0);
				l_underlyingOriginalUnoDocumentInXStorable2: XStorable2 = cast (XStorable2, l_underlyingFileOpeningUnoDispatcherInXSynchronousDispatch.dispatchWithReturnValue (l_originalFileUrlInURL, l_unoDocumentOpeningProperties))
				l_hasSucceeded: bool = False
				if l_underlyingOriginalUnoDocumentInXStorable2 is not None:
					try:
						l_underlyingOriginalUnoDocumentInXStorable2.storeToURL (l_targetFileUrl, l_unoDocumentStoringProperties)
						l_hasSucceeded = True
					except (Exception) as l_exception:
						raise l_exception
					cast (XCloseable, l_underlyingOriginalUnoDocumentInXStorable2).close (False)
				else:
					sys.stdout.write ("### The original file: '{0:s}' cannot be opened.\n".format (l_originalFileUrl))
					sys.stdout.flush ()
		~

if __name__ == "__main__":
	Test1Test.main (sys.argv)

Lenard
Can I just copy the code? Do I have to change something?

Special-Student-7
For the minimum requirements, you do not have to change anything, supposing that your code has already gotten a UNO objects context and has appropriately set the URLs and the file-storing-filter name.

Lenard
Has my code already gotten a UNO objects context?

Special-Student-7
I do not know. . . . If it has not, it can get it according to an article (for just or for a scrupulous way).

Jane
Where can I find my desired filter name?

Special-Student-7
The possible filter names are cited in the concept part of this article.

Jane
Well, what is that code doing?

Lenard
It's converting a file.

Jane
I know that. How is it doing so?

Special-Student-7
The code is doing exactly what I said in a section of the concept part of this article: open the original file, store the opened document in the target format, and close the opened document.

'dispatchWithReturnValue' is opening the original file, 'storeToURL' is storing the opened document, and 'close' is closing the opened document.

'l_underlyingFileOpeningUnoDispatcherInXSynchronousDispatch' is a UNO dispatcher and does not need to be gotten per file.

2: On Properties for Opening or Storing

Special-Student-7
As is seen in the above code, you can set some properties for opening the file. In fact, these are the specifications of the possible properties, where the datum types are Python types (instead of UNO types).

The name	The datum type	The description
ReadOnly	bool	whether the opened document cannot be written back into the original file: 'True'-> cannot, 'False'-> can
Hidden	bool	whether the opened document is hidden: 'True'-> hidden, 'False'-> not hidden
OpenNewView	bool	whether the document is opened in a new view: 'True'-> in a new view, 'False'-> not in a new view
Silent	bool	whether the document is opened silently: 'True'-> silently, 'False'-> not silently
Password	str	the opening password

A property you may be interested with is the opening password.

Jane
Hmm, also files that are protected by opening passwords can be converted programmatically.

Lenard
I will make the file owners tell me the passwords.

Jane
You shouldn't; it's a passwords theft; let the users input the passwords into the program.

Special-Student-7
Please note that UNO 'string' does not officially accept 'None', so, if there is no opening password, the property itself should not be specified, although 'None' might happen to be accepted.

And please note that 'ReadOnly' does not mean that the opened document cannot be modified, but means that the document cannot be written back into the original file, so, it does not need to be 'False' in order for the document to be tweaked and written into a new file.

You can set some properties also for storing the document. In fact, these are the specifications of the possible properties, where the datum types are Python types (instead of UNO types).

The name	The datum type	The description
FilterName	str	the filter name
FilterData	depends on the case	the filter-specific data for some filters: not always but often a list of 'com.sun.star.beans.PropertyValue'
FilterOptions	str	the filter-specific data for some filters
AsTemplate	bool	whether the document is stored as a template: 'True'-> as a template, 'False'-> not as a template
Author	str	the author name
DocumentTitle	str	the document title
EncryptionData	com.sun.star.beans.NamedValue []	the encryption data for some filters
Password	str	the opening password for some filters
CharacterSet	str	the characters set
Version	int	the version number
Comment	str	the version description
Overwrite	bool	whether the file is overwritten: 'True' -> overwritten, 'False' -> not overwritten
ComponentData	depends on the case	some filter-specific data: not always but often a list of 'com.sun.star.beans.NamedValue'
ModifyPasswordInfo	depends on the case	the editing password data for some filters

The opening password is really nothing but encryption data; the reason why there are both of 'EncryptionData' and 'Password' is that some filters use one and some other filters use the other; for details, please refer to another article.

The situation concerning setting an editing password is rather complicated: the editing password has to be hashed by a format-specific algorithm and the hash has to be set into a format-specific location, which is not necessarily inside the file-storing properties. . . . The details are explained in an article for OpenDocument files, for Microsoft binary files, and for Office Open XML files.

Lenard
I don't like editing password; why shouldn't I edit the file?

Jane
Because you aren't authorized to change the contents.

Special-Student-7
Well, editing password is not so effective for anti-tampering and is principally for not accidentally changing the contents, but anyway, as my workable sample has implemented the logic, you can use it, if you will.

Jane
I don't know what exactly I can do with those "filter-specific" properties like 'FilterData' and 'FilterOptions'.

Special-Student-7
That cannot be explained in this article for the general logic of file conversion: please refer to some other articles like for PDF files and for CSV files.

3: On Tweaking the Document

Special-Student-7
The target file may not be satisfactorily controlled only via the file-storing properties.

Then, you can tweak the opened document.

Lenard
"tweak"? Isn't it tampering?

Special-Student-7
No, not particularly: the modifications are meant to go only into the target file; the original file is meant to be intact.

Lenard
"meant" . . ., so, the programmer can surreptitiously tamper original files!

Special-Student-7
If the programmer is malicious, tampering files is not the only thing you have to worry about. Well, you can take a measure like granting the program only the reading privilege for the original files.

Lenard
Oh, don't teach that to anybody!

Jane
He wants to tamper files as the programmer.

Special-Student-7
In the above code, 'l_underlyingOriginalUnoDocumentInXStorable2' represents the UNO object of the opened document, via which you can tweak the opened document.

Lenard
How exactly can I tweak the document?

Special-Student-7
That is of course document-specific and requirement-specific, and please refer to some other articles like for tweaking page properties of word processor documents (here and here) and for tweaking page properties of spread sheets documents (here and here).

4: Here Is a Workable Sample

Special-Student-7
A workable sample can be downloaded from here.

The sample program takes as an argument the path of a JSON file (in fact, it can take multiple JSON file paths) that contains file conversion orders, and fulfills the orders.

The specifications of the JSON file are like these.

@JSON Source Code

[
	{Disabled: %whether this record is disabled or not -> 'true' or 'false'%, 
	OriginalFileUrl: %the original file URL, in which any operating system environment variable can be used like '%{HOME}'%, 
	TargetFileUrl: %the target file URL, in which any operating system environment variable can be used like '%{HOME}'%, 
	FileStoringFilterName: %the file-storing-filter name%, 
	OpeningPassword: %the opening password for the original file%, 
	Title: %the target file title%, 
	EncryptingPassword: %the encrypting password for the target file%, 
	EditingPassword: %the editing password for the target file%, 
	CertificateIssuerName: %the certificate issuer name for signing the target file%, 
	CertificatePassword: %the certificate password for signing the target file%, 
	SignatureTimeStampAuthorityUrl: %the time stamp authority URL for signing the target file%, 
	DocumentTailorNames: [%a document tailor name%, . . .], 
	AllSpreadSheetsAreExported: %whether all the spread sheets are exported or not: 'true' or 'false'%, 
	TargetFileNamingRule: %the naming rule for target CSV files: '0'-> each CSV file is named with the sheet index, '1'-> each CSV file is named with the sheet name%, 
	CsvItemsDelimiterCharacterCode: %the character code of the CSV items delimiter: '44'-> ',' for example%, 
	CsvTextItemQuotationCharacterCode: %the character code of the CSV text item quotation: '34'-> '"' for example%, 
	CsvCharactersEncodingCode: %the CSV characters encoding code: '76' -> UTF-8, '65535' -> UCS-2, '65534' -> UCS-4, '11' -> US-ASCII, '69' -> EUC_JP, '64' -> SHIFT_JIS%, 
	CsvAllTextItemsAreQuoted: %whether all the CSV text items are quoted or not: 'true' or 'false'%, 
	CsvContentsAreExportedAsShown: %whether the CSV contents are exported as shown on the sheets: 'true' or 'false'%, 
	CsvFormulaeThemselvesAreExported: %whether the sheet cell formulae themselves are exported into the CSV files: 'true' or 'false'%, 
	HiddenSpreadSheetsAreExported: %whether also the hidden spread sheets are exported%}, 
	. . .
	{. . .}
]

Jane
What is "tailor", by the way?

Special-Student-7
Any tailor is an object that is tasked with tweaking the document in a specific way.

Jane
So, I have to create it myself, don't I?

Special-Student-7
I have prepared a few sample tailors, but yes, you will have to create one yourself if you have a specific need.

Jane
And do I specify the class full name?

Special-Student-7
No, you need to specify the object name: there may be some multiple tailors from a class, constructed with different parameters.

Jane
Huh? What do you mean by "the object name"? The variable name?

Special-Student-7
No. The key value for the object in the dictionary in which the tailors are registered. . . . Do not worry; we will talk about it in a later subsection.

Anyway, if your conversion orders come as Web requests for example, you could form an in-memory JSON record from each Web request and pass the JSON record into the 'executeConversionOrder' method, which takes such a JSON record.

Lenard
So, I can use the method, even if the program receives the orders in any way.

Special-Student-7
If you need additional capabilities in the JSON record, you can modify the 'executeConversionOrder' method together with the 'FileConversionOrderExtendedJsonDatumParseEventsHandler' JSON parse-events-handler.

Jane
Hmm.

Special-Student-7
I have made the sample to accept multiple orders files, which are processed concurrently, in order to demonstrate that conversions can be done concurrently in this optimal way.

4-1: Building the Sample Project

Special-Student-7
How to build the sample project is explained in an article.

Especially, note that the projects contain also implementations in some other programming languages, which you, a Python programmer, will probably not need; how to ignore such unnecessary code is explained there.

The main project is 'filesConverter'.

4-2: Executing the Sample Program

Special-Student-7
Before we execute the sample program, we have to have started a LibreOffice or Apache OpenOffice instance so that it accepts connections from clients, as we have learned how in a previous article.

Stage Direction
Special-Student-7 starts a LibreOffice instance with the port number 2002.

Special-Student-7
We need a conversion orders JSON file (in the specifications cited above), and we will use the sample JSON file included in the archive file as 'filesConverter/data/FileConversionOrders.json', but at least the file URLs inside it have to be adjusted.

We can execute the sample program like this with the current directory positioned at the sample project directory.

@cmd Source Code

gradle i_executePythonExecutableFileTask -Pi_mainModuleName="theBiasPlanet.filesConverter.programs.FilesConverterConsoleProgram" -Pi_commandLineArguments="\"socket,host=localhost,port=2002,tcpNoDelay=1;urp;StarOffice.ComponentContext\" \"data/FileConversionOrders.json\""

Stage Direction
Special-Student-7 executes the command in the terminal with the current directory positioned at the sample project directory.

Special-Student-7
We can just change the host name from 'localhost' if the LibreOffice or Apache OpenOffice instance is in a remote host, if some firewalls do not block the communication, of course.

And you can specify multiple JSON files like this.

@cmd Source Code

gradle i_executePythonExecutableFileTask -Pi_mainModuleName="theBiasPlanet.filesConverter.programs.FilesConverterConsoleProgram" -Pi_commandLineArguments="\"socket,host=localhost,port=2002,tcpNoDelay=1;urp;StarOffice.ComponentContext\" \"data/FileConversionOrders.json\" \"data/FileConversionOrders2.json\""

4-3: Understanding the Sample Code

Special-Student-7
Note that the sample code uses my utility projects ('coreUtilitiesToBeDisclosed' and 'unoUtilitiesToBeDisclosed'), which include code that is not directly related to the work of file conversion; please understand that it is not so easy to excise only irrelevant parts of the utility projects, because the code is intertwined.

The file conversion logic is in the 'convertFile' method (except for converting to CSV files) or the 'convertSpreadSheetsDocumentFileToCsvFiles' method (for converting to CSV files) of the 'theBiasPlanet.unoUtilities.filesConverting.FilesConverter' class, which (the method) takes the document tailors and the file-storing properties, among some other things.

In order to install a new document tailor, you have to have the class as a 'theBiasPlanet.unoUtilities.documentsHandling.UnoDocumentTailor' child and register an instance of the class with its unique name into the 'l_unoDocumentTailorNameToUnoDocumentTailorMap' variable of the 'theBiasPlanet.filesConverter.programs.FilesConverterConsoleProgram' class.

References

The Bias Planet
Exploiting LibreOffice or OpenOffice
Exposing the Java Programming Language
Exposing C++
Exposing Python
Exposing C#
Exposing Gradle
Exposing Git
To Disentangle Confusing Terms or Discourses
Definitions and Propositions
School Mathematics from Higher Viewpoints
UNO Dispatch Commands
Information Tables

2019-11-24