Filesystems in ACCRE ==================== Guidelines of database model creation ------------------------------------- For any new file system added to ACCRE, the following things must be defined in regards to admintools and automation around the filesystem: 1. A table in the models.py file that will describe the new filesystem. The filesystem table must contain the following columns: a. Identifiers that can uniquely identify a filesystem entry. For example, for gpfs, "filesystem", "name", and "fileset'' define a unique entry. Similarly for panfs, it would be the combination of "bladeset" and “name”. b. The filesystem table must contain a foreign key to groups that are existing under ACCRE where applicable. It is not applicable in case of user quotas and system related quotas, the group should be defaulted to accre. c. The table should also have the following items if applicable: i. soft_quota: BigInteger ii. hard_quota: BigInteger iii. path: str iv. user_path: str v. active: bool vi. join_date: datetime 2. A usage table with followings: i. soft_quota: BigInteger ii. hard_quota: BigInteger iii. space_used: BigInteger iv. path: str v. user_path: str vi. user, optional: str. Only applicable for paths such as /home/username, /nobackup/userspace/username vi. last_updated, automatic: datetime. Updates itself automatically to reflect when it was recently changed vii. join_date: datetime Filesystems in ACCRE -------------------- The `basefilesystem.py` file provides the class GenericFileSystemMixin. It is meant to be a guideline for development of filesystem handlers in ACCRE. Implementation for FutureVolume ------------------------------- :: from typing import Optional from pydantic import BaseModel from sqlalchemy import Column, DateTime, JSON, String, Table, MetaData, BigInteger from accre.util import convert_byte_unit from accre.filesystem.basefilesystem import GenericFileSystemMixin admin_meta = MetaData() FUTURE_VOLUME = Table('future_volume', admin_meta, Column('name', String, primary_key=True, nullable=False), Column('path', String, primary_key=True, nullable=False), Column('soft_quota', BigInteger, nullable=False), Column('hard_quota', BigInteger, nullable=False), Column('join_date', DateTime) ) FUTURE_USAGE = Table('future_usage', admin_meta, Column('name', String, primary_key=True, nullable=False), Column('path', String, primary_key=True, nullable=False), Column('usage', BigInteger, nullable=False), Column('extra_info', JSON, nullable=True), ) class FutureVolumeModel(BaseModel): """Future Volume Model""" name: str path: str soft_quota: str hard_quota: str class FutureVolumeModelPartial(BaseModel): """Future Volume Model Partial""" name: Optional[str] = "" path: Optional[str] = "" soft_quota: Optional[str] = "" hard_quota: Optional[str] = "" class FutureVolume(GenericFileSystemMixin): """ Test future volume """ database_table = FUTURE_VOLUME usage_table = FUTURE_USAGE data_model = FutureVolumeModel data_model_partial = FutureVolumeModelPartial data_holder = {} simlinks = {} deployed_simlinks = {} def process_fields_db(self, validated_volume_input, insert=False) -> dict: return_dict = {**validated_volume_input.dict()} if isinstance(validated_volume_input.hard_quota, str): return_dict["hard_quota"] = int(convert_byte_unit(validated_volume_input.hard_quota, target='B')) if isinstance(validated_volume_input.soft_quota, str): return_dict["soft_quota"] = int(convert_byte_unit(validated_volume_input.soft_quota, target='B')) return return_dict def process_usage_fields(self, validated_volume_input, usage_details: dict): return_dict = { "name": validated_volume_input.name, "path": validated_volume_input.path, "extra_info": usage_details.get("extra_info", {}) } if "usage" in usage_details: return_dict["usage"] = usage_details.get("usage") if isinstance(usage_details.get("usage"), str): return_dict["usage"] = int(convert_byte_unit(usage_details.get("usage"), target='B')) return return_dict def get_volume_fs(self, validated_volume_input, update_usage=True, timeout=600): value = self.data_holder.get(validated_volume_input.name) if update_usage: self.add_or_update_usage_table(validated_volume_input, value) return value def modify_volume_fs(self, validated_volume_input): self.data_holder[validated_volume_input.name] = { **self.data_holder[validated_volume_input.name], **validated_volume_input.dict(), "usage": "1G", } return True, "", self.data_holder[validated_volume_input.name] def add_volume_fs(self, validated_volume_input): self.data_holder[validated_volume_input.name] = { **validated_volume_input.dict(), "usage": "1G", "extra_info": {} } return True, "", self.data_holder[validated_volume_input.name] def remove_volume_fs(self, volume_input): self.data_holder.pop(volume_input.name) return True, "" def delete_symlink(self, validated_volume_input): self.simlinks.pop(validated_volume_input.name) return True def create_symlink(self, validated_volume_input): self.simlinks[validated_volume_input.name] = True return True def deploy_symlinks(self): self.deployed_simlinks = {**self.simlinks} return True def get_volumes_fs(self): return list(self.data_holder.values()) def get_users_usage_fs(self): return [ { "name": "user1", "path": "/home/user1", "usage": "1G", "soft_quota": "1G" , "hard_quota": "2G", "extra_info": { "files_used": 10, "files_soft_quota": 100, "files_hard_quota": 1000, } }, { "name": "user2", "path": "/home/user2", "usage": "1G", "soft_quota": "1G" , "hard_quota": "2G", "extra_info": { "files_used": 10, "files_soft_quota": 100, "files_hard_quota": 1000, } } ] Mismatched volumes and groups ----------------------------- There are certain volumes in panfs that don't exactly map to groups. For instance: `/data/cqs` maps to group `h_cqs` instead of `cqs` To mitigate such mismappings, something like this can be done: :: from accre.filesystem.panfs.main import PanasasFileSystem fs = PanasasFileSystem() fs.mis_matched_volume_groups = { "/data/cqs": "h_cqs", "/nobackup/testing": "fe_accre_lab" } filesystem ========== .. automodule:: accre.filesystem.basefilesystem :members: :show-inheritance: filesystems.panfs ================= .. automodule:: accre.filesystem.panfs.main :members: :show-inheritance: filesystems.utils ================= .. automodule:: accre.filesystem.panfs.utils :members: :show-inheritance: