Skip to content
Snippets Groups Projects
Commit e771e66b authored by Julien David's avatar Julien David
Browse files

clone project

parents
No related branches found
No related tags found
No related merge requests found
Showing
with 802 additions and 0 deletions
LICENSE 0 → 100644
RDOS is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
RDOS is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with RDOS. If not, see <https://www.gnu.org/licenses/>.
\ No newline at end of file
# RDOS: Random Discrete Object Suite
To launch the rdos_service:
- cd software
- docker stack deploy --compose-file docker-compose.yml rdos_service
\ No newline at end of file
#########################################################################################
# DEAMON #
#########################################################################################
# # base image
# FROM alpine:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# # variable
# ENV AUTHOR Docker
# # Instalation de dépendances
# RUN apk update
# RUN apk add python3
# RUN apk add py-pip
# RUN pip install mysql-connector-python
# RUN pip install get_docker_secret
# RUN pip install docker
# ADD deamon/ /usr/local/rdos_deamon
# ADD database/ /usr/local/database/
# # change working directory
# WORKDIR /usr/local/rdos_deamon/
# # run command
# ENTRYPOINT ["python3","rdos_deamon.py"]
# # # DO NOT FORGET to share the Docker socket of the host with the container
# # #docker run -v /var/run/docker.sock:/var/run/docker.sock -v /home/david/rdos/jobs/:/usr/local/jobs --env working_dir=/usr/local/ -tid rdos_deamon
#########################################################################################
# CLEANER #
#########################################################################################
# # base image
FROM alpine:latest
# maintainer
LABEL maintainer="david@lipn.fr"
# variable
ENV AUTHOR Docker
# Instalation de dépendances
RUN apk update
RUN apk add python3
ADD cleaner/ /usr/local/rdos_cleaner
# change working directory
WORKDIR /usr/local/rdos_cleaner
# run command
ENTRYPOINT ["python3","rdos_cleaner.py"]
# # How to run me
# # docker run -v /home/david/rdos/jobs/:/usr/local/jobs --env working_dir=/usr/local/ -tid rdos_cleaner
# #######################################################################################
# # LAUNCHER #
# #######################################################################################
# # base image
# FROM alpine:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# # variable
# ENV AUTHOR Docker
# #Instalation de dépendances
# RUN apk update
# RUN apk add python3
# RUN apk add py-pip
# RUN pip install mysql-connector-python
# RUN pip install get_docker_secret
# ADD . /usr/local/rdos
# # change working directory
# WORKDIR /usr/local/rdos/
# # run command
# #RUN python3
# ENTRYPOINT ["python3","launcher.py"]
#######################################################################################
# REGAL #
#######################################################################################
# # base image
# FROM ubuntu:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# #Instalation de dépendances
# RUN apt-get update
# RUN apt-get install -y python3 python3-pip
# RUN pip3 install mysql-connector-python
# RUN pip3 install get_docker_secret
# RUN DEBIAN_FRONTEND="noninteractive" apt-get -y install tzdata
# RUN apt-get install -y g++ make
# RUN apt-get install -y libglibmm-2.4-1v5
# #RUN apt-cache search libxml++ | grep xml++
# RUN apt-get install -y libxml++2.6-2v5 libxml++2.6-dev
# ADD generators/PREGA/ /usr/local/regal
# ADD launcher/ /usr/local/regal
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/regal
# # run command
# RUN make
# ENTRYPOINT ["python3","launcher.py"]
#######################################################################################
# DASS #
#######################################################################################
# #base image
# FROM alpine:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# # variable
# ENV AUTHOR Docker
# #Instalation de dépendances
# RUN apk update
# RUN apk add python2
# RUN apk add python3
# RUN apk add py-pip
# RUN pip install mysql-connector-python
# RUN pip install get_docker_secret
# ADD generators/DAAS/ /usr/local/daas
# ADD launcher/ /usr/local/daas
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/daas
# # run command
# ENTRYPOINT ["python3","launcher.py"]
#######################################################################################
# WEAKLY DIRECTED #
#######################################################################################
# # base image
# FROM ubuntu:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# # variable
# ENV AUTHOR Docker
# #Instalation de dépendances
# RUN apt-get update
# RUN apt-get install -y --no-install-recommends python3 python3-pip
# RUN pip3 install mysql-connector-python
# RUN pip3 install get_docker_secret
# ADD generators/Axel/weakly_directed /usr/local/wd/weakly_directed
# ADD launcher/ /usr/local/wd/
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/wd/
# # run command
# ENTRYPOINT ["python3","launcher.py"]
#######################################################################################
# WEAKLY PRUDENT #
#######################################################################################
# # base image
# FROM ubuntu:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# # variable
# ENV AUTHOR Docker
# #Instalation de dépendances
# RUN apt-get update
# RUN apt-get install -y --no-install-recommends python3 python3-pip
# RUN pip3 install mysql-connector-python
# RUN pip3 install get_docker_secret
# ADD generators/Axel/weakly_prudent /usr/local/wp/weakly_prudent
# ADD launcher/ /usr/local/wp/
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/wp/
# # run command
# ENTRYPOINT ["python3","launcher.py"]
#######################################################################################
# NEWTONGF #
#######################################################################################
# # base image
# FROM alpine:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# # variable
# ENV AUTHOR Docker
# #Instalation de dépendances
# RUN apk update
# RUN apk add python3
# RUN apk add py-pip
# RUN pip install mysql-connector-python
# RUN pip install get_docker_secret
# ADD generators/Carine_Mapple /usr/local/newtongf/Carine_Mapple
# ADD generators/Carine_Mapplesys /usr/local/newtongf/Carine_Mapplesys
# ADD generators/Carine_Mapplesys.mpl /usr/local/newtongf/Carine_Mapplesys.mpl
# ADD launcher/ /usr/local/newtongf/
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/newtongf
# # # run command
# # RUN make
# # ENTRYPOINT ["python3","launcher.py"]
# CMD /bin/ash
#######################################################################################
# NFA GNERATOR #
#######################################################################################
# # base image
# FROM alpine:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# # variable
# ENV AUTHOR Docker
# #Instalation de dépendances
# RUN apk update
# RUN apk add python3
# RUN apk add py-pip
# RUN pip install mysql-connector-python
# RUN pip install get_docker_secret
# RUN apk add libc-dev make
# RUN apk add g++
# ADD generators/NFAGenerator/ /usr/local/nfagenerator
# ADD launcher/ /usr/local/nfagenerator
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/nfagenerator/
# # run command
# RUN make
# ENTRYPOINT ["python3","launcher.py"]
#######################################################################################
# GenRGenS #
#######################################################################################
# # base image
# FROM alpine:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# #Instalation de dépendances
# RUN apk update
# RUN apk add openjdk7-jre
# RUN apk add python3
# RUN apk add py-pip
# RUN pip install mysql-connector-python
# RUN pip install get_docker_secret
# ADD generators/GenRGenS2.1/ /usr/local/GenRGenS
# ADD launcher/ /usr/local/GenRGenS
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/GenRGenS
# # run command
# ENTRYPOINT ["python3","launcher.py"]
#######################################################################################
# Arbogen #
#######################################################################################
# # base image
# FROM alpine:latest
# # maintainer
# LABEL maintainer="david@lipn.fr"
# #Instalation de dépendances
# RUN apk update
# RUN apk add ocaml ocamlbuild libc-dev make
# RUN apk add python3
# RUN apk add py-pip
# RUN pip install mysql-connector-python
# RUN pip install get_docker_secret
# ADD generators/arbogen-master/ /usr/local/arbogen-master
# ADD launcher/ /usr/local/arbogen-master
# ADD database/ /usr/local/database/
# ADD mailer /usr/local/mailer/
# # change working directory
# WORKDIR /usr/local/arbogen-master
# # run command
# RUN make
# RUN mv bin/arbogen.byte .
# ENTRYPOINT ["python3","launcher.py"]
import os
from os import listdir
import time
# Routine that deletes old directories
def clean_old_directories(path: str, max_age:int):
directories_path = os.path.join(path,"jobs")
for f in listdir(directories_path):
if f != "index.html":
job_path = os.path.join(directories_path,f)
st = os.stat(job_path)
age = time.time() - st.st_mtime
if (age > max_age): #by default, after two days, directory is deleted
for file in listdir(job_path):
os.remove(os.path.join(job_path, file))
os.rmdir(job_path)
print("Dossier ", job_path, "nettoyé")
working_dir = "/home/david/rdos/"
if os.getenv("working_dir") != None:
working_dir = os.getenv("working_dir")
max_age = 172800 # two days in seconds
if os.getenv("max_age") != None :
working_dir = int(os.getenv("max_age"))
while 1:
clean_old_directories(working_dir, max_age)
time.sleep(1)
File added
import mysql.connector
from get_docker_secret import get_docker_secret
import os
# Exception thrown when no password has been defined to connect to the database
class NoPasswordDefineError(Exception):
"""No secret containing a password has been found"""
pass
__rdos_password__ = None
# Procedures that returns an externally defined password to conect to the database
# First tries to check the docker secrets, then an environment variable.
# Throws NoPasswordDefineError if no password is found
# Returns a password
def get_password():
if __rdos_password__ == None:
password = get_docker_secret('rdos_secret')
if password == None:
if 'rdos_secret' in os.environ:
password = os.environ['rdos_secret']
print("Password stored in a non secured way, please make your code safer")
else:
raise NoPasswordDefineError
return password
return __rdos_password__
# Connect to the local MYSQL database on lipn-rdos
def database_connection():
mydb = mysql.connector.connect(
host="192.168.90.101",
user="rdos",
password = get_password(),
database="rdos"
)
return mydb
def update_database(url:str, job_id: str, before:str, after:str):
database = database_connection()
mycursor = database.cursor(buffered=False)
with open('../database/templates/update_success.txt', 'r') as file:
update = file.read()
update = update.replace('{{before}}', before).replace('{{after}}', after)
update = update.replace('{{url}}', url)
update = update.replace('{{job_id}}', job_id)
print(update)
mycursor.execute(update)
database.commit()
print(mycursor.rowcount, "record(s) affected")
return mycursor.rowcount
# Get a row in the Job table that hasn't been computed yet.
def get_job(database):
sql_get_new_job = "SELECT jobs.id,parametersJSON,directory,email,tool,jobs.idGenerator,outputFormat,command from jobs inner join generators on jobs.idGenerator=generators.id where status='10'"
mycursor = database.cursor(buffered=True)
mycursor.execute(sql_get_new_job)
myresult = mycursor.fetchone()
mycursor.close()
return myresult
# Tells the database that the previously selected job will be taken care of by the deamon
# Returns 1 if no other deamon selected the job first, 0 otherwwise.
def apply_for_job(database, job_id: str):
mycursor = database.cursor(buffered=False)
update = "UPDATE jobs set status='100' where jobs.id='"+job_id+"' and status='10'"
mycursor.execute(update)
database.commit()
return mycursor.rowcount
# Returns a dictionnary containing each generator's parameters as a key. The value associated to each key is "is this parameter's value written in the comand line or in a separated file
# in which case the value is the filename.
def getParameters(generators: dict):
database = database_connection()
parameters_basefile = {}
parameters_default = {}
for generator in generators:
parameters_basefile[generator] = {}
parameters_default[generator] = {}
sql_get_new_job = "SELECT name,basefile,defaultValue from parameters inner join generators on parameters.idGenerator=generators.id where generators.tool = '"+generator+"'"
mycursor = database.cursor(buffered=False)
mycursor.execute(sql_get_new_job)
myresult = mycursor.fetchall()
mycursor.close()
for result in myresult:
parameters_basefile[generator][result[0]] = result[1]
parameters_default[generator][result[0]] = result[1]
database.disconnect()
return (parameters_basefile, parameters_default)
UPDATE jobs set status='100000', timeExecuted = '{{before}}', timeFinished = '{{after}}', url = '{{url}}' where jobs.id = '{{job_id}}'
File added
{
"generators":[
{"name":"Regal", "path":"/home/david/rdos/softwares/PREGA", "image":"regal"},
{"name":"DAAS", "path":"/home/david/rdos/softwares/DAAS", "image":"daas" },
{"name":"Weakly Directed", "path":"/home/david/rdos/softwares/Axel", "image":"weakly_directed"},
{"name":"Weakly Prudent", "path":"/home/david/rdos/softwares/Axel", "image":"weakly_prudent" },
{"name":"GenRGenS", "path":"/home/david/rdos/softwares/GenRGenS2.1", "image":"genrgens"},
{"name":"NewtonGF", "path":"/home/david/rdos/softwares/Carine_Mapple/", "image":"newtongf"},
{"name":"Arbogen", "path":"/home/david/rdos/softwares/arbogen-master/bin", "image":"arbogen"},
{"name":"NFAGenerator", "path":"/home/david/rdos/softwares/NFAGenerator", "image":"nfagenerator"}
]
}
import json
# Some JSON string created by the web site are impossible to parse by python-json library
# Thus, we replace special characters
# 'paramaters' is the JSON string
# Returns a dictionnary containing the key-values of the JSON string.
def trim_and_load_json(parameters:str):
parameters = parameters.replace('\r','')
parameters = parameters.replace('\n',' ')
return json.loads(parameters)
# Create the commands that will be executed by the launcher container.
# Replaces parameters names by the values selected by the user.
def create_command(parameters_info: tuple, tool: str, path: str,command: str, parameters:str , job_dir:str):
(parameters_basefile, parameters_default) = parameters_info
param_dict = trim_and_load_json(parameters)
command = command.replace("{path}", ".")
for param in param_dict:
value = param_dict[param]
if len(value) == 0: # If the value is empty, replaces it by a default value
value = parameters_default[tool][param]
#value = value.replace('\\\\n', ' ')
if len(parameters_basefile[tool][param]) == 0: # Check whether the parameter should be given through a file or not.
command = command.replace('{'+param+'}', value)
else:
command = command.replace('{'+param+'}', os.path.join(volume_point,job_dir,parameters_basefile[tool][param]))
with open(os.path.join(working_dir, job_dir,parameters_basefile[tool][param]), 'w') as file:
file.write(value)
file.close()
command += " > "+os.path.join(volume_point,job_dir,"result.txt")
print(command)
return command
import mysql.connector
from get_docker_secret import get_docker_secret
import time, datetime
import os
import sys
sys.path.append(os.path.realpath('../'))
from database.database import *
import docker
import json
from rdos_command import *
################################### FILESYSTEM INTERACTIONS ###################################
# Returns a dictionnary containing the generator's name as a key and it's associated image as a value
def getGenerators(path: str):
with open(path, 'r') as file:
data = file.read()
file.close()
obj = json.loads(data)
generators_path = {}
for generator in obj['generators']:
generators_path[str(generator["name"])] = ( str(generator["path"]), str(generator["image"]))
return generators_path
#Creates a directory where the job's result will be written, but also some jobs parameters.
def create_directory(directory_name: str):
try:
os.stat(directory_name)
except:
os.mkdir(directory_name)
print("Directory", directory_name," created")
# Create the commands that will be executed by the launcher container.
# Replaces parameters names by the values selected by the user.
def create_command(parameters_info: tuple, tool: str, path: str,command: str, parameters:str , job_dir:str):
(parameters_basefile, parameters_default) = parameters_info
parameters = parameters.replace('\r','')
parameters = parameters.replace('\n',' ')
param_dict = json.loads(parameters)
command = command.replace("{path}", ".")
for param in param_dict:
value = param_dict[param]
if len(value) == 0: # If the value is empty, replaces it by a default value
value = parameters_default[tool][param]
#value = value.replace('\\\\n', ' ')
if len(parameters_basefile[tool][param]) == 0: # Check whether the parameter should be given through a file or not.
command = command.replace('{'+param+'}', value)
else:
command = command.replace('{'+param+'}', os.path.join(volume_point,job_dir,parameters_basefile[tool][param]))
with open(os.path.join(working_dir, job_dir,parameters_basefile[tool][param]), 'w') as file:
file.write(value)
file.close()
command += " > "+os.path.join(volume_point,job_dir,"result.txt")
print(command)
return command
################################### MAIN PROGRAM ###################################
working_dir = "/home/david/rdos/"
if os.getenv("working_dir") != None:
working_dir = os.getenv("working_dir")
volume_point = "/usr/local/"
generators = getGenerators("generators.json")
client = docker.from_env()
parameters_info = getParameters(generators)
while 1:
database = database_connection()
result = get_job(database)
if (not result == None):
(job_id, parametersJSON, directory, email, tool, idGenerator,outputFormat,command) = result
if tool in generators:
if apply_for_job(database, job_id) == 1 :
create_directory(directory.replace("./", working_dir))
print(directory)
command = create_command(parameters_info, tool, generators[tool][0], command, parametersJSON, directory)
client.containers.run(generators[tool][1], detach=True, environment= {"job_id":job_id, "command":command, "directory":directory, "email":email, "rdos_secret":get_password()}, volumes = {'/home/david/rdos/jobs/': {'bind': '/usr/local/jobs', 'mode': 'rw'}})
print(command," appelée sur un nouveau conteneur ", generators[tool][1])
else:
print("Tool ",tool," Unknown")
database.disconnect()
time.sleep(1)
version: '3.5'
secrets:
rdos_secret:
name: rdos_secret
services:
nginx:
image: nginx:latest
container_name : composed_rdos_nginx
ports:
- "7070:80"
volumes:
- /home/david/rdos/jobs:/usr/share/nginx/html/jobs
cleaner:
image: rdos_cleaner:latest
container_name : composed_rdos_cleaner
volumes:
- /home/david/rdos/jobs/:/usr/local/jobs
environment:
- working_dir=/usr/local/
deamon:
image: rdos_deamon:latest
container_name : composed_rdos_deamon
depends_on:
- nginx
- cleaner
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/david/rdos/jobs/:/usr/local/jobs
environment:
- working_dir=/usr/local/
- rdos_secret=rdos2013
\ No newline at end of file
File added
{
"url":"http://lipn.fr/rdos2/jobs/"
}
import os
import sys
sys.path.append(os.path.realpath('../'))
from mailer.mail_templates import *
from database.database import *
import time
from datetime import datetime
import json
def getConfiguration(path: str):
with open(path, 'r') as file:
data = file.read()
return json.loads(data)
def create_directory(directory_name: str):
try:
os.stat(directory_name)
except:
os.mkdir(directory_name)
print("Directory", directory_name," created")
job_id = os.getenv("job_id")
command= os.getenv("command")
directory= os.getenv("directory")
email= os.getenv("email")
configuration = getConfiguration("configuration.json")
if (not job_id == None) and (not command == None) :
create_directory(directory.replace("./","/usr/local/"))
print("Execution de la commande ", command)
before = datetime.fromtimestamp(time.time())
os.system(command)
after = datetime.fromtimestamp(time.time())
print("Tâche effectuée en ", after-before)
url = os.path.join( configuration["url"], command.split(">")[1].split("jobs/")[1] )
update_database(url, job_id, str(before), str(after))
send_mail(email,job_id)
File added
#!/usr/bin/python
import smtplib
import socket
def success(email: str, jobid:str):
with open("../mailer/templates/rdos_success.txt") as file:
message = file.read().replace("{{email}}",email).replace("{{jobid}}", jobid)
return message
def failed(email: str, error_message:str):
with open("../mailer/templates/rdos_failure.txt") as file:
message = file.read().replace("{{email}}",email).replace("{{error_message}}", error_message)
return message
def send_mail(email: str, job_id:str) :
if len(email) > 0:
try:
smtpObj = smtplib.SMTP('mail.lipn.univ-paris13.fr')
smtpObj.sendmail('rdos@lipn.univ-paris13.fr',
email, success(email, job_id))
print("Successfully sent email to: ", email)
return 1
except smtplib.SMTPException:
print("Error: unable to send email")
return -1
except socket.timeout:
print("Time Out, could not connect to server")
return -1
else:
print("Email not sent (Empty email address)")
return 0
From: RDOS <rdos@lipn.univ-paris13.fr>
To: RDOS User <{{email}}>
Subject: RDOS: Did you order some discrete objects?
Thank you for using Rdos.
Unfortunately, your order could not be delivered because of the following reason:
{{error_message}}
From: RDOS <rdos@lipn.univ-paris13.fr>
To: RDOS User <{{email}}>
Subject: RDOS: Did you order some discrete objects?
Thank you for using Rdos.
Please download your results at the following address:
http://lipn.univ-paris13.fr/rdos/viewjob.php?lang=en&css=bioinfo&jobid={{jobid}}
Note that the files will be deleted in 48 hours.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment